| 2012-11-21
      
      § | 
    
  | 18:45 | <AaronSchulz> | Copying over all math/timeline files to nas1 that are not already there for DR | [production] | 
            
  | 18:34 | <AaronSchulz> | prepared all wikivoyage file directories and containers | [production] | 
            
  | 17:41 | <Reedy> | Running scap-recompile on all hosts in mediawiki-installation group | [production] | 
            
  | 16:38 | <mutante> | powering on analytics1007 | [production] | 
            
  | 16:29 | <mutante> | reinstalling srv284 "last lucid standing" | [production] | 
            
  | 16:24 | <mark> | Takeback by nas1001-a completed, snapmirror relationships back in sync | [production] | 
            
  | 16:15 | <mark> | Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback | [production] | 
            
  | 16:08 | <mutante> | powercycling srv284 | [production] | 
            
  | 16:05 | <mutante> | remove known troublemakers srv266 and srv284 from dsh group "apaches" | [production] | 
            
  | 15:50 | <mark> | Initiated takeover of nas1001-a to nas1001-b | [production] | 
            
  | 15:49 | <mark> | nas1001-b back up and running | [production] | 
            
  | 14:46 | <mark> | Halted nas1001-b for NIC upgrade | [production] | 
            
  | 09:18 | <notpeter> | repooling mw20-mw34 | [production] | 
            
  | 07:50 | <notpeter> | repooling mw70-mw74 | [production] | 
            
  | 07:21 | <apergos> | restarted lucene search on search14 | [production] | 
            
  | 03:01 | <notpeter> | depooling mw70-mw74 for upgrade to precise | [production] | 
            
  | 03:01 | <notpeter> | depooling mw20-mw34 for upgrade to precise | [production] | 
            
  | 02:45 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012 | [production] | 
            
  | 02:24 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012 | [production] | 
            
  | 02:17 | <K4-713> | updated payments cluster to 6ea97a5cbc06 | [production] | 
            
  | 00:40 | <logmsgbot_> | preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 00:37 | <logmsgbot_> | preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 00:06 | <logmsgbot_> | maxsem synchronized php-1.21wmf4/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | [production] | 
            
  | 00:03 | <logmsgbot_> | maxsem synchronized php-1.21wmf3/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | [production] | 
            
  | 00:03 | <notpeter> | depooling mw66-mw99 for upgrade to apache | [production] | 
            
  | 00:03 | <notpeter> | depooling mw53-mw49 for upgrade to apache | [production] | 
            
  
    | 2012-11-20
      
      § | 
    
  | 23:52 | <logmsgbot_> | maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 23:35 | <K4-713> | updated payments cluster to b5a80e04086c3a7 | [production] | 
            
  | 23:32 | <binasher> | depooled srv203 | [production] | 
            
  | 23:00 | <logmsgbot_> | reedy synchronized php-1.21wmf4/ | [production] | 
            
  | 22:56 | <logmsgbot_> | reedy synchronized php-1.21wmf3/ | [production] | 
            
  | 22:45 | <logmsgbot_> | kaldari synchronized wmf-config/InitialiseSettings.php  'disabling experimental flickr uploading on commons until some bugs are fixed' | [production] | 
            
  | 22:44 | <logmsgbot_> | maxsem Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 22:35 | <logmsgbot_> | asher synchronized wmf-config/InitialiseSettings.php  'now only 30 wikis that need wgOldChangeTagsIndex = true, changing default to false' | [production] | 
            
  | 21:02 | <notpeter> | depooling mw17-mw19 and mw50-mw54 for upgrade to precise | [production] | 
            
  | 20:59 | <notpeter> | depooling mw62-mw65 for upgrade to precise | [production] | 
            
  | 20:37 | <Jeff_Green> | switching payments back to eqiad | [production] | 
            
  | 20:23 | <binasher> | running change_tag index migrations on the 757 wikis with <7k ct rows | [production] | 
            
  | 20:09 | <logmsgbot_> | kaldari synchronized wmf-config/InitialiseSettings.php  'turning on flickr uploading for sysops on Commons' | [production] | 
            
  | 19:42 | <apergos> | scribbled over /dev/sda partition table on ms-be7 so I could watch the install break, left in shell form installer | [production] | 
            
  | 16:14 | <notpeter> | depooling srv281-srv289 for upgrades to precise | [production] | 
            
  | 09:57 | <hashar> | Upgrading Jenkins plugin xUnit which introduce a potential back compatibility issue <failedIfNotNew> renamed to <failIfNotNew>. Will update configs files. | [production] | 
            
  | 09:37 | <hashar> | restarting Jenkins | [production] | 
            
  | 09:22 | <hashar> | upgrading Jenkins plugins | [production] | 
            
  | 09:18 | <logmsgbot_> | nikerabbit synchronized wmf-config/InitialiseSettings.php  'i18n deploy' | [production] | 
            
  | 08:15 | <logmsgbot_> | tstarling Finished syncing Wikimedia installation... : | [production] | 
            
  | 07:33 | <paravoid> | rebooting ms-be5, strange df output for / | [production] | 
            
  | 07:01 | <logmsgbot_> | tstarling Started syncing Wikimedia installation... : | [production] | 
            
  | 06:37 | <logmsgbot_> | tstarling synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 06:36 | <logmsgbot_> | tstarling synchronized wmf-config/InitialiseSettings.php | [production] |