| 2012-11-21
      
      § | 
    
  | 23:47 | <mutante> | tmp. depooling and rebooting a few mw5x servers for kernel upgrades, one by one | [production] | 
            
  | 23:37 | <logmsgbot_> | preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 23:34 | <logmsgbot_> | preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 22:58 | <mutante> | disabled swap on kaulen per Tim's advice | [production] | 
            
  | 22:55 | <mutante> | bugzilla back up | [production] | 
            
  | 22:50 | <mutante> | powercycling kaulen | [production] | 
            
  | 22:25 | <logmsgbot_> | asher synchronized wmf-config/throttle.php  'increased api limit for [[bugzilla:42319|bug 42319]]' | [production] | 
            
  | 21:51 | <mutante> | installing package upgrades in the mw55-mw99 range | [production] | 
            
  | 21:17 | <logmsgbot_> | preilly Finished syncing Wikimedia installation... : update zero rated mobile access | [production] | 
            
  | 20:56 | <Ryan_Lane> | deploying Change If5f6bc33: ([[bugzilla:42334|bug 42334]]) on labsconsole | [production] | 
            
  | 20:47 | <Ryan_Lane> | applying changes to fix no creds issue when setting preferences on labsconsole | [production] | 
            
  | 20:47 | <logmsgbot_> | preilly Started syncing Wikimedia installation... : update zero rated mobile access | [production] | 
            
  | 20:34 | <logmsgbot_> | preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 20:34 | <logmsgbot_> | preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 20:00 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.21wmf4 | [production] | 
            
  | 19:24 | <mutante> | installing package upgrades in the srv193-srv199 range | [production] | 
            
  | 18:55 | <mutante> | re-signing srv284 on puppetmaster, package installs... | [production] | 
            
  | 18:45 | <AaronSchulz> | Copying over all math/timeline files to nas1 that are not already there for DR | [production] | 
            
  | 18:34 | <AaronSchulz> | prepared all wikivoyage file directories and containers | [production] | 
            
  | 17:41 | <Reedy> | Running scap-recompile on all hosts in mediawiki-installation group | [production] | 
            
  | 16:38 | <mutante> | powering on analytics1007 | [production] | 
            
  | 16:29 | <mutante> | reinstalling srv284 "last lucid standing" | [production] | 
            
  | 16:24 | <mark> | Takeback by nas1001-a completed, snapmirror relationships back in sync | [production] | 
            
  | 16:15 | <mark> | Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback | [production] | 
            
  | 16:08 | <mutante> | powercycling srv284 | [production] | 
            
  | 16:05 | <mutante> | remove known troublemakers srv266 and srv284 from dsh group "apaches" | [production] | 
            
  | 15:50 | <mark> | Initiated takeover of nas1001-a to nas1001-b | [production] | 
            
  | 15:49 | <mark> | nas1001-b back up and running | [production] | 
            
  | 14:46 | <mark> | Halted nas1001-b for NIC upgrade | [production] | 
            
  | 09:18 | <notpeter> | repooling mw20-mw34 | [production] | 
            
  | 07:50 | <notpeter> | repooling mw70-mw74 | [production] | 
            
  | 07:21 | <apergos> | restarted lucene search on search14 | [production] | 
            
  | 03:01 | <notpeter> | depooling mw70-mw74 for upgrade to precise | [production] | 
            
  | 03:01 | <notpeter> | depooling mw20-mw34 for upgrade to precise | [production] | 
            
  | 02:45 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012 | [production] | 
            
  | 02:24 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012 | [production] | 
            
  | 02:17 | <K4-713> | updated payments cluster to 6ea97a5cbc06 | [production] | 
            
  | 00:40 | <logmsgbot_> | preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 00:37 | <logmsgbot_> | preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | [production] | 
            
  | 00:06 | <logmsgbot_> | maxsem synchronized php-1.21wmf4/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | [production] | 
            
  | 00:03 | <logmsgbot_> | maxsem synchronized php-1.21wmf3/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | [production] | 
            
  | 00:03 | <notpeter> | depooling mw66-mw99 for upgrade to apache | [production] | 
            
  | 00:03 | <notpeter> | depooling mw53-mw49 for upgrade to apache | [production] |