| 
      
        2012-11-20
      
      ยง
     | 
  
    
  | 23:52 | 
  <logmsgbot_> | 
  maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 23:35 | 
  <K4-713> | 
  updated payments cluster to b5a80e04086c3a7 | 
  [production] | 
            
  | 23:32 | 
  <binasher> | 
  depooled srv203 | 
  [production] | 
            
  | 23:00 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/ | 
  [production] | 
            
  | 22:56 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf3/ | 
  [production] | 
            
  | 22:45 | 
  <logmsgbot_> | 
  kaldari synchronized wmf-config/InitialiseSettings.php  'disabling experimental flickr uploading on commons until some bugs are fixed' | 
  [production] | 
            
  | 22:44 | 
  <logmsgbot_> | 
  maxsem Started syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 22:35 | 
  <logmsgbot_> | 
  asher synchronized wmf-config/InitialiseSettings.php  'now only 30 wikis that need wgOldChangeTagsIndex = true, changing default to false' | 
  [production] | 
            
  | 21:02 | 
  <notpeter> | 
  depooling mw17-mw19 and mw50-mw54 for upgrade to precise | 
  [production] | 
            
  | 20:59 | 
  <notpeter> | 
  depooling mw62-mw65 for upgrade to precise | 
  [production] | 
            
  | 20:37 | 
  <Jeff_Green> | 
  switching payments back to eqiad | 
  [production] | 
            
  | 20:23 | 
  <binasher> | 
  running change_tag index migrations on the 757 wikis with <7k ct rows | 
  [production] | 
            
  | 20:09 | 
  <logmsgbot_> | 
  kaldari synchronized wmf-config/InitialiseSettings.php  'turning on flickr uploading for sysops on Commons' | 
  [production] | 
            
  | 19:42 | 
  <apergos> | 
  scribbled over /dev/sda partition table on ms-be7 so I could watch the install break, left in shell form installer | 
  [production] | 
            
  | 16:14 | 
  <notpeter> | 
  depooling srv281-srv289 for upgrades to precise | 
  [production] | 
            
  | 09:57 | 
  <hashar> | 
  Upgrading Jenkins plugin xUnit which introduce a potential back compatibility issue <failedIfNotNew> renamed to <failIfNotNew>. Will update configs files. | 
  [production] | 
            
  | 09:37 | 
  <hashar> | 
  restarting Jenkins | 
  [production] | 
            
  | 09:22 | 
  <hashar> | 
  upgrading Jenkins plugins | 
  [production] | 
            
  | 09:18 | 
  <logmsgbot_> | 
  nikerabbit synchronized wmf-config/InitialiseSettings.php  'i18n deploy' | 
  [production] | 
            
  | 08:15 | 
  <logmsgbot_> | 
  tstarling Finished syncing Wikimedia installation... : | 
  [production] | 
            
  | 07:33 | 
  <paravoid> | 
  rebooting ms-be5, strange df output for / | 
  [production] | 
            
  | 07:01 | 
  <logmsgbot_> | 
  tstarling Started syncing Wikimedia installation... : | 
  [production] | 
            
  | 06:37 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/CommonSettings.php | 
  [production] | 
            
  | 06:36 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 06:36 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/filebackend.php | 
  [production] | 
            
  | 06:34 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf4/extensions/Score | 
  [production] | 
            
  | 06:33 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf3/extensions/Score | 
  [production] | 
            
  | 05:54 | 
  <Ryan_Lane> | 
  rebooting ssl3001 | 
  [production] | 
            
  | 05:19 | 
  <Ryan_Lane> | 
  depooling ssl3001 | 
  [production] | 
            
  | 05:19 | 
  <Ryan_Lane> | 
  repooling ssl3002 (upgrade complete) | 
  [production] | 
            
  | 05:14 | 
  <paravoid> | 
  deploying squid config for score | 
  [production] | 
            
  | 05:11 | 
  <Ryan_Lane> | 
  rebooting ssl3002 | 
  [production] | 
            
  | 04:48 | 
  <Ryan_Lane> | 
  depooling ssl3002 for upgrade to precise | 
  [production] | 
            
  | 04:48 | 
  <Ryan_Lane> | 
  repooling ssl3003 (upgrade finished) | 
  [production] | 
            
  | 04:42 | 
  <Ryan_Lane> | 
  rebooting ssl3003 (upgraded via ssh, not console) | 
  [production] | 
            
  | 04:25 | 
  <notpeter> | 
  repooling srv258-srv280 | 
  [production] | 
            
  | 04:04 | 
  <Ryan_Lane> | 
  repooling ssl1 (upgrade complete) | 
  [production] | 
            
  | 03:52 | 
  <Tim> | 
  on brewster: reduced rotate count for squid logs to zero and ran logrotate -f | 
  [production] | 
            
  | 03:44 | 
  <Tim> | 
  on brewster: root partition is full, removing some useless squid logs | 
  [production] | 
            
  | 02:52 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf3) at Tue Nov 20 02:52:09 UTC 2012 | 
  [production] | 
            
  | 02:28 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf4) at Tue Nov 20 02:28:20 UTC 2012 | 
  [production] | 
            
  | 02:10 | 
  <notpeter> | 
  depooling srv296-srv301 for upgrades to precise | 
  [production] | 
            
  | 02:08 | 
  <notpeter> | 
  repooling srv290-srv295 | 
  [production] | 
            
  | 01:21 | 
  <Ryan_Lane> | 
  depooling ssl1 | 
  [production] | 
            
  | 01:21 | 
  <Ryan_Lane> | 
  repooling ssl2 (upgrade complete) | 
  [production] | 
            
  | 01:10 | 
  <Ryan_Lane> | 
  depooling ssl3003 to upgrade to precise | 
  [production] | 
            
  | 01:02 | 
  <MaxSem> | 
  Scap hung and had to be aborted. Since what was being deployed wasn't enabled, no clusters were harmed. | 
  [production] | 
            
  | 00:46 | 
  <Ryan_Lane> | 
  all eqiad https hosts upgraded to precise | 
  [production] | 
            
  | 00:46 | 
  <Ryan_Lane> | 
  repooling ssl1001 (upgrade complete) | 
  [production] | 
            
  | 00:23 | 
  <Ryan_Lane> | 
  depooling ssl2 for upgrade to precise | 
  [production] |