| 2012-07-05
      
      § | 
    
  | 23:54 | <Ryan_Lane> | force running puppet on formey and manganese, since a config change is involved, it's going to restart | [production] | 
            
  | 23:45 | <Ryan_Lane> | restarting gerrit on manganese | [production] | 
            
  | 23:38 | <Ryan_Lane> | upgrading gerrit on formey | [production] | 
            
  | 23:21 | <Ryan_Lane> | updating database for gerrit | [production] | 
            
  | 23:09 | <Ryan_Lane> | upgrading gerrit on manganese | [production] | 
            
  | 23:08 | <Ryan_Lane> | stopping gerrit on manganese and disabling puppet | [production] | 
            
  | 23:07 | <Ryan_Lane> | stopping gerrit on formey and disabling puppet | [production] | 
            
  | 22:47 | <preilly> | synchronized wmf-config/mobile.php  'add subdomain check' | [production] | 
            
  | 22:34 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 22:17 | <reedy> | synchronized wmf-config/  'Tidying config for randomrootpage' | [production] | 
            
  | 21:56 | <reedy> | synchronized wmf-config/  'Strike 2' | [production] | 
            
  | 21:50 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'revert' | [production] | 
            
  | 21:48 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Random root page enabled everywhere but wikipedias' | [production] | 
            
  | 21:47 | <kaldari> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 21:34 | <kaldari> | Started syncing Wikimedia installation... : | [production] | 
            
  | 21:18 | <kaldari> | synchronized wmf-config/InitialiseSettings.php  'Bumping AFTv5 lottery percentage to 20% of en.wiki' | [production] | 
            
  | 21:15 | <LeslieCarr> | powercycling unresponsive mw1116 | [production] | 
            
  | 21:09 | <maplebed> | added new ms-be pmtpa hosts to DNS | [production] | 
            
  | 20:03 | <preilly> | synchronized php-1.20wmf6/extensions/MobileFrontend  'fix cache issue with token' | [production] | 
            
  | 19:14 | <Jeff_Green> | *somebody* purged binary logs on blondel | [production] | 
            
  | 18:58 | <preilly> | synchronized php-1.20wmf6/extensions/MobileFrontend  'fix cache issue' | [production] | 
            
  | 18:54 | <mark> | Upgraded pybal on all precise LVS servers | [production] | 
            
  | 18:43 | <mark> | Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository | [production] | 
            
  | 17:41 | <mutante> | powercycling db1048, mw1136, mw1046 | [production] | 
            
  | 17:32 | <midom> | synchronized wmf-config/InitialiseSettings.php  'reenabling db40' | [production] | 
            
  | 17:30 | <mutante> | more powercycling and upgrading: mw1036, mw1134, mw1043 .. | [production] | 
            
  | 17:26 | <mutante> | powercycling db1009,db1010 | [production] | 
            
  | 17:22 | <mutante> | powercycling db1027,db1028 | [production] | 
            
  | 17:14 | <mutante> | powercycling db1013 | [production] | 
            
  | 16:50 | <mutante> | powercycling downed db1015 | [production] | 
            
  | 16:44 | <mutante> | powercycling and upgrading more mw10xx servers, 1017,1023,1025 ... | [production] | 
            
  | 16:27 | <mutante> | mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot | [production] | 
            
  | 16:18 | <mutante> | powercycling mw1002 | [production] | 
            
  | 15:37 | <mutante> | argon back up with new kernel,mysql,grub,.. looks happy afaict | [production] | 
            
  | 15:33 | <mutante> | argon (limesurvey) fscked, dist-upgrading | [production] | 
            
  | 15:28 | <mutante> | powercycling argon | [production] | 
            
  | 15:26 | <hashar> | updated Jenkins configuration on gallium : Updating f407ebe..4b669b9 | [production] | 
            
  | 14:08 | <cmjohnson1> | HUME replacing disk 0 | [production] | 
            
  | 14:08 | <midom> | synchronized wmf-config/InitialiseSettings.php  'disabling parser cache for now' | [production] | 
            
  | 11:54 | <mark> | Inserted new pybal_1.02 package into APT distribution precise-wikimedia | [production] | 
            
  | 11:17 | <mark> | Installed new pybal snapshot build for testing on lvs1005 | [production] | 
            
  | 11:13 | <mutante> | add missing Russian locales on singer, run localegen, run ru.planet update | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.20wmf6) at Thu Jul  5 02:26:45 UTC 2012 | [production] |