| 2012-02-15
      
      § | 
    
  | 19:05 | <RoanKattouw> | Rerunning l10nupdate by hand to hopefully fix CDB problems | [production] | 
            
  | 19:02 | <RoanKattouw> | Fixing permissions for php-1.18/cache/l10n on all apaches as root | [production] | 
            
  | 18:56 | <RoanKattouw> | Running sync-l10nupdate by hand to see what kind of perms errors I get, first for 1.18 then for 1.19 | [production] | 
            
  | 17:01 | <RobH> | cadmium setup for wikimania video transcoding | [production] | 
            
  | 16:37 | <RobH> | forgot to log, carbon resumed service normally | [production] | 
            
  | 16:23 | <RobH> | carbon halted, allows login and freezes on password entry, rebooting | [production] | 
            
  | 15:53 | <RobH> | os install on cadmium | [production] | 
            
  | 15:49 | <RobH> | updating dns for cadmium | [production] | 
            
  | 15:06 | <reedy> | synchronized php-1.19/extensions/SpamBlacklist/SpamBlacklistHooks.php  '[[rev:111543|r111543]]' | [production] | 
            
  | 13:12 | <hashar> | synchronized wmf-config/codereview.php | [production] | 
            
  | 11:32 | <mutante> | sync-apache / graceful not logged anymore by logmsgbot ? | [production] | 
            
  | 10:06 | <mutante> | office.wm now forces https (in a less broken way;) (remnant.conf) | [production] | 
            
  | 09:42 | <mutante> | made a new change to remnant.conf and synced apaches in a fresh attempt to fix office.wm redirect | [production] | 
            
  | 08:59 | <Nirvanchik> | test | [production] | 
            
  | 08:13 | <mutante> | all search boxes had /home: Stale NFS file handle.. remounting | [production] | 
            
  | 08:03 | <mutante> | remounted /home on search6, started lsearchd | [production] | 
            
  | 02:37 | <Tim> | on kaulen: re-enabled jsonrpc.cgi and reduced MaxClients from 500 to 100 | [production] | 
            
  | 02:18 | <LocalisationUpdate> | failed (1.19) at Wed Feb 15 02:18:04 UTC 2012 | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.18) at Wed Feb 15 02:18:04 UTC 2012 | [production] | 
            
  | 01:18 | <reedy> | synchronized php-1.19/extensions/Gadgets/Gadgets_body.php | [production] | 
            
  | 01:09 | <reedy> | synchronized php-1.19/extensions/Gadgets/Gadgets_body.php | [production] | 
            
  | 00:38 | <maplebed> | deployed new thumb_handler.php with ETag header added in to ms5 | [production] | 
            
  | 00:11 | <reedy> | synchronized php-1.19/extensions/Contest/Contest.php  'revert live hack' | [production] | 
            
  
    | 2012-02-14
      
      § | 
    
  | 23:52 | <reedy> | synchronized wmf-config/flaggedrevs.php  'Enable FR like dewiki on test2wiki' | [production] | 
            
  | 23:43 | <LeslieCarr> | allowed ipv6 pim on edge routers in the US | [production] | 
            
  | 23:25 | <reedy> | synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js  '[[rev:111506|r111506]]' | [production] | 
            
  | 23:21 | <LeslieCarr> | modifying "martian" blocks on cr2-eqiad to allow newly allocated ip ranges | [production] | 
            
  | 23:19 | <reedy> | synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js  '[[rev:111506|r111506]]' | [production] | 
            
  | 23:18 | <reedy> | synchronized php-1.19/includes  '[[rev:111486|r111486]]' | [production] | 
            
  | 22:53 | <reedy> | synchronized php-1.19/includes  '[[rev:111486|r111486]]' | [production] | 
            
  | 22:46 | <reedy> | ran sync-common-all | [production] | 
            
  | 22:45 | <maplebed> | deployed squid config to upload to send all thumbnail traffic to ms5 instead of swift | [production] | 
            
  | 22:33 | <Reedy> | running ddsh -F5 -cM -g mediawiki-installation 'sudo -u mwdeploy rm -rf /usr/local/apache/common-local/php-1.17' | [production] | 
            
  | 22:26 | <Reedy> | Removing php-1.17 from fenari | [production] | 
            
  | 22:10 | <binasher> | restarted the 1.19 schema migration script - it's going to hit the just rotated s3 (db34), s2 (db30), s7 (db16), and s4 (db31) ex-masters before resuming s5 (db55) and all s6/s1 slaves | [production] | 
            
  | 22:06 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Setting wmfUseRevSha1Columns' | [production] | 
            
  | 21:55 | <Ryan_Lane> | restarting ircecho on spence | [production] | 
            
  | 21:48 | <binasher> | new s4 master pos - MASTER_LOG_FILE='db22-bin.000030', MASTER_LOG_POS=964208442 | [production] | 
            
  | 21:48 | <asher> | synchronized wmf-config/db.php  'setting s4 to writeable, new master is db22, db31 still out' | [production] | 
            
  | 21:47 | <asher> | synchronized wmf-config/db.php  'setting s4 to read-only, switching master to db22' | [production] | 
            
  | 21:39 | <asher> | synchronized wmf-config/db.php  'returning db30 to s3, going to wait til after schema migrations to upgrade to lucid/new-mysql' | [production] | 
            
  | 21:36 | <binasher> | new s2-master pos - MASTER_LOG_FILE='db13-bin.000278', MASTER_LOG_POS=599752853 | [production] | 
            
  | 21:36 | <asher> | synchronized wmf-config/db.php  'setting s2 to writeable, db13 is new master' | [production] | 
            
  | 21:35 | <asher> | synchronized wmf-config/db.php  'setting s2 to read-only, switching master to db13' | [production] | 
            
  | 21:29 | <asher> | synchronized wmf-config/db.php  'db34 upgrading, returning to s3' | [production] | 
            
  | 21:15 | <binasher> | new s3 master position - MASTER_LOG_FILE='db39-bin.000550', MASTER_LOG_POS=63238699 | [production] | 
            
  | 21:15 | <asher> | synchronized wmf-config/db.php  'returning s3 to writeable, db39 is the new master' | [production] | 
            
  | 21:13 | <asher> | synchronized wmf-config/db.php  'setting s3 to read-only, switching master to db39' | [production] | 
            
  | 21:02 | <asher> | synchronized wmf-config/db.php  'returning db16 after upgrading mysql' | [production] | 
            
  | 20:58 | <binasher> | new s7 repl position - MASTER_LOG_FILE='db37-bin.000285', MASTER_LOG_POS=865712092 | [production] |