| 
      
        2012-11-21
      
      §
     | 
  
    
  | 23:47 | 
  <mutante> | 
  tmp. depooling and rebooting a few mw5x servers for kernel upgrades, one by one | 
  [production] | 
            
  | 23:37 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 23:34 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 22:58 | 
  <mutante> | 
  disabled swap on kaulen per Tim's advice | 
  [production] | 
            
  | 22:55 | 
  <mutante> | 
  bugzilla back up | 
  [production] | 
            
  | 22:50 | 
  <mutante> | 
  powercycling kaulen | 
  [production] | 
            
  | 22:25 | 
  <logmsgbot_> | 
  asher synchronized wmf-config/throttle.php  'increased api limit for [[bugzilla:42319|bug 42319]]' | 
  [production] | 
            
  | 21:51 | 
  <mutante> | 
  installing package upgrades in the mw55-mw99 range | 
  [production] | 
            
  | 21:17 | 
  <logmsgbot_> | 
  preilly Finished syncing Wikimedia installation... : update zero rated mobile access | 
  [production] | 
            
  | 20:56 | 
  <Ryan_Lane> | 
  deploying Change If5f6bc33: ([[bugzilla:42334|bug 42334]]) on labsconsole | 
  [production] | 
            
  | 20:47 | 
  <Ryan_Lane> | 
  applying changes to fix no creds issue when setting preferences on labsconsole | 
  [production] | 
            
  | 20:47 | 
  <logmsgbot_> | 
  preilly Started syncing Wikimedia installation... : update zero rated mobile access | 
  [production] | 
            
  | 20:34 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 20:34 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 20:00 | 
  <logmsgbot_> | 
  reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.21wmf4 | 
  [production] | 
            
  | 19:24 | 
  <mutante> | 
  installing package upgrades in the srv193-srv199 range | 
  [production] | 
            
  | 18:55 | 
  <mutante> | 
  re-signing srv284 on puppetmaster, package installs... | 
  [production] | 
            
  | 18:45 | 
  <AaronSchulz> | 
  Copying over all math/timeline files to nas1 that are not already there for DR | 
  [production] | 
            
  | 18:34 | 
  <AaronSchulz> | 
  prepared all wikivoyage file directories and containers | 
  [production] | 
            
  | 17:41 | 
  <Reedy> | 
  Running scap-recompile on all hosts in mediawiki-installation group | 
  [production] | 
            
  | 16:38 | 
  <mutante> | 
  powering on analytics1007 | 
  [production] | 
            
  | 16:29 | 
  <mutante> | 
  reinstalling srv284 "last lucid standing" | 
  [production] | 
            
  | 16:24 | 
  <mark> | 
  Takeback by nas1001-a completed, snapmirror relationships back in sync | 
  [production] | 
            
  | 16:15 | 
  <mark> | 
  Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback | 
  [production] | 
            
  | 16:08 | 
  <mutante> | 
  powercycling srv284 | 
  [production] | 
            
  | 16:05 | 
  <mutante> | 
  remove known troublemakers srv266 and srv284 from dsh group "apaches" | 
  [production] | 
            
  | 15:50 | 
  <mark> | 
  Initiated takeover of nas1001-a to nas1001-b | 
  [production] | 
            
  | 15:49 | 
  <mark> | 
  nas1001-b back up and running | 
  [production] | 
            
  | 14:46 | 
  <mark> | 
  Halted nas1001-b for NIC upgrade | 
  [production] | 
            
  | 09:18 | 
  <notpeter> | 
  repooling mw20-mw34 | 
  [production] | 
            
  | 07:50 | 
  <notpeter> | 
  repooling mw70-mw74 | 
  [production] | 
            
  | 07:21 | 
  <apergos> | 
  restarted lucene search on search14 | 
  [production] | 
            
  | 03:01 | 
  <notpeter> | 
  depooling mw70-mw74 for upgrade to precise | 
  [production] | 
            
  | 03:01 | 
  <notpeter> | 
  depooling mw20-mw34 for upgrade to precise | 
  [production] | 
            
  | 02:45 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012 | 
  [production] | 
            
  | 02:24 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012 | 
  [production] | 
            
  | 02:17 | 
  <K4-713> | 
  updated payments cluster to 6ea97a5cbc06 | 
  [production] | 
            
  | 00:40 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 00:37 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 00:06 | 
  <logmsgbot_> | 
  maxsem synchronized php-1.21wmf4/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | 
  [production] | 
            
  | 00:03 | 
  <logmsgbot_> | 
  maxsem synchronized php-1.21wmf3/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | 
  [production] | 
            
  | 00:03 | 
  <notpeter> | 
  depooling mw66-mw99 for upgrade to apache | 
  [production] | 
            
  | 00:03 | 
  <notpeter> | 
  depooling mw53-mw49 for upgrade to apache | 
  [production] |