| 
      
        2012-11-21
      
      §
     | 
  
    
  | 16:29 | 
  <mutante> | 
  reinstalling srv284 "last lucid standing" | 
  [production] | 
            
  | 16:24 | 
  <mark> | 
  Takeback by nas1001-a completed, snapmirror relationships back in sync | 
  [production] | 
            
  | 16:15 | 
  <mark> | 
  Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback | 
  [production] | 
            
  | 16:08 | 
  <mutante> | 
  powercycling srv284 | 
  [production] | 
            
  | 16:05 | 
  <mutante> | 
  remove known troublemakers srv266 and srv284 from dsh group "apaches" | 
  [production] | 
            
  | 15:50 | 
  <mark> | 
  Initiated takeover of nas1001-a to nas1001-b | 
  [production] | 
            
  | 15:49 | 
  <mark> | 
  nas1001-b back up and running | 
  [production] | 
            
  | 14:46 | 
  <mark> | 
  Halted nas1001-b for NIC upgrade | 
  [production] | 
            
  | 09:18 | 
  <notpeter> | 
  repooling mw20-mw34 | 
  [production] | 
            
  | 07:50 | 
  <notpeter> | 
  repooling mw70-mw74 | 
  [production] | 
            
  | 07:21 | 
  <apergos> | 
  restarted lucene search on search14 | 
  [production] | 
            
  | 03:01 | 
  <notpeter> | 
  depooling mw70-mw74 for upgrade to precise | 
  [production] | 
            
  | 03:01 | 
  <notpeter> | 
  depooling mw20-mw34 for upgrade to precise | 
  [production] | 
            
  | 02:45 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf3) at Wed Nov 21 02:45:42 UTC 2012 | 
  [production] | 
            
  | 02:24 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf4) at Wed Nov 21 02:24:47 UTC 2012 | 
  [production] | 
            
  | 02:17 | 
  <K4-713> | 
  updated payments cluster to 6ea97a5cbc06 | 
  [production] | 
            
  | 00:40 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 00:37 | 
  <logmsgbot_> | 
  preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess  'update post deploy' | 
  [production] | 
            
  | 00:06 | 
  <logmsgbot_> | 
  maxsem synchronized php-1.21wmf4/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | 
  [production] | 
            
  | 00:03 | 
  <logmsgbot_> | 
  maxsem synchronized php-1.21wmf3/extensions/MobileFrontend  'https://gerrit.wikimedia.org/r/#/c/34440/' | 
  [production] | 
            
  | 00:03 | 
  <notpeter> | 
  depooling mw66-mw99 for upgrade to apache | 
  [production] | 
            
  | 00:03 | 
  <notpeter> | 
  depooling mw53-mw49 for upgrade to apache | 
  [production] | 
            
  
    | 
      
        2012-11-20
      
      §
     | 
  
    
  | 23:52 | 
  <logmsgbot_> | 
  maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 23:35 | 
  <K4-713> | 
  updated payments cluster to b5a80e04086c3a7 | 
  [production] | 
            
  | 23:32 | 
  <binasher> | 
  depooled srv203 | 
  [production] | 
            
  | 23:00 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/ | 
  [production] | 
            
  | 22:56 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf3/ | 
  [production] | 
            
  | 22:45 | 
  <logmsgbot_> | 
  kaldari synchronized wmf-config/InitialiseSettings.php  'disabling experimental flickr uploading on commons until some bugs are fixed' | 
  [production] | 
            
  | 22:44 | 
  <logmsgbot_> | 
  maxsem Started syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 22:35 | 
  <logmsgbot_> | 
  asher synchronized wmf-config/InitialiseSettings.php  'now only 30 wikis that need wgOldChangeTagsIndex = true, changing default to false' | 
  [production] | 
            
  | 21:02 | 
  <notpeter> | 
  depooling mw17-mw19 and mw50-mw54 for upgrade to precise | 
  [production] | 
            
  | 20:59 | 
  <notpeter> | 
  depooling mw62-mw65 for upgrade to precise | 
  [production] | 
            
  | 20:37 | 
  <Jeff_Green> | 
  switching payments back to eqiad | 
  [production] | 
            
  | 20:23 | 
  <binasher> | 
  running change_tag index migrations on the 757 wikis with <7k ct rows | 
  [production] | 
            
  | 20:09 | 
  <logmsgbot_> | 
  kaldari synchronized wmf-config/InitialiseSettings.php  'turning on flickr uploading for sysops on Commons' | 
  [production] | 
            
  | 19:42 | 
  <apergos> | 
  scribbled over /dev/sda partition table on ms-be7 so I could watch the install break, left in shell form installer | 
  [production] | 
            
  | 16:14 | 
  <notpeter> | 
  depooling srv281-srv289 for upgrades to precise | 
  [production] | 
            
  | 09:57 | 
  <hashar> | 
  Upgrading Jenkins plugin xUnit which introduce a potential back compatibility issue <failedIfNotNew> renamed to <failIfNotNew>. Will update configs files. | 
  [production] | 
            
  | 09:37 | 
  <hashar> | 
  restarting Jenkins | 
  [production] | 
            
  | 09:22 | 
  <hashar> | 
  upgrading Jenkins plugins | 
  [production] | 
            
  | 09:18 | 
  <logmsgbot_> | 
  nikerabbit synchronized wmf-config/InitialiseSettings.php  'i18n deploy' | 
  [production] | 
            
  | 08:15 | 
  <logmsgbot_> | 
  tstarling Finished syncing Wikimedia installation... : | 
  [production] | 
            
  | 07:33 | 
  <paravoid> | 
  rebooting ms-be5, strange df output for / | 
  [production] | 
            
  | 07:01 | 
  <logmsgbot_> | 
  tstarling Started syncing Wikimedia installation... : | 
  [production] | 
            
  | 06:37 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/CommonSettings.php | 
  [production] | 
            
  | 06:36 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 06:36 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/filebackend.php | 
  [production] | 
            
  | 06:34 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf4/extensions/Score | 
  [production] | 
            
  | 06:33 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf3/extensions/Score | 
  [production] | 
            
  | 05:54 | 
  <Ryan_Lane> | 
  rebooting ssl3001 | 
  [production] |