| 
      
        2012-07-06
      
      §
     | 
  
    
  | 15:54 | 
  <reedy> | 
  synchronized images/sul/meta.png  | 
  [production] | 
            
  | 15:45 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 15:36 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 15:21 | 
  <cmjohnson1> | 
  ms-be5 removing disk0 to replace with good disk | 
  [production] | 
            
  | 14:38 | 
  <RobHalsell> | 
  authdns-update to correct typo in mgmt dns entry | 
  [production] | 
            
  | 13:22 | 
  <mark> | 
  Inserted new pybal 1.04 package in the precise-wikimedia APT repository, and upgraded all precise LVS servers | 
  [production] | 
            
  | 11:31 | 
  <tstarling> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 11:25 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 11:20 | 
  <Tim> | 
  running scap to delete docroot files per I48502d90 and Ie3afd137 | 
  [production] | 
            
  | 08:58 | 
  <mutante> | 
  dist-upgrading (unused) db10xx servers | 
  [production] | 
            
  | 07:43 | 
  <Tim> | 
  also trusted-xff.phps | 
  [production] | 
            
  | 07:40 | 
  <Tim> | 
  removing some untracked junk from /home/wikipedia/common | 
  [production] | 
            
  | 07:29 | 
  <mutante> | 
  continue to restart and upgrade downed mw10xx servers | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.20wmf6) at Fri Jul  6 02:26:22 UTC 2012 | 
  [production] | 
            
  | 01:35 | 
  <binasher> | 
  updated squid redirector to cover wiki(quotes|books|versity) | 
  [production] | 
            
  | 00:13 | 
  <Ryan_Lane> | 
  restarting gerrit | 
  [production] | 
            
  | 00:09 | 
  <Ryan_Lane> | 
  force running puppet on manganese, it'll restart gerrit | 
  [production] | 
            
  
    | 
      
        2012-07-05
      
      §
     | 
  
    
  | 23:54 | 
  <Ryan_Lane> | 
  force running puppet on formey and manganese, since a config change is involved, it's going to restart | 
  [production] | 
            
  | 23:45 | 
  <Ryan_Lane> | 
  restarting gerrit on manganese | 
  [production] | 
            
  | 23:38 | 
  <Ryan_Lane> | 
  upgrading gerrit on formey | 
  [production] | 
            
  | 23:21 | 
  <Ryan_Lane> | 
  updating database for gerrit | 
  [production] | 
            
  | 23:09 | 
  <Ryan_Lane> | 
  upgrading gerrit on manganese | 
  [production] | 
            
  | 23:08 | 
  <Ryan_Lane> | 
  stopping gerrit on manganese and disabling puppet | 
  [production] | 
            
  | 23:07 | 
  <Ryan_Lane> | 
  stopping gerrit on formey and disabling puppet | 
  [production] | 
            
  | 22:47 | 
  <preilly> | 
  synchronized wmf-config/mobile.php  'add subdomain check' | 
  [production] | 
            
  | 22:34 | 
  <reedy> | 
  synchronized wmf-config/  | 
  [production] | 
            
  | 22:17 | 
  <reedy> | 
  synchronized wmf-config/  'Tidying config for randomrootpage' | 
  [production] | 
            
  | 21:56 | 
  <reedy> | 
  synchronized wmf-config/  'Strike 2' | 
  [production] | 
            
  | 21:50 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'revert' | 
  [production] | 
            
  | 21:48 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Random root page enabled everywhere but wikipedias' | 
  [production] | 
            
  | 21:47 | 
  <kaldari> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 21:34 | 
  <kaldari> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 21:18 | 
  <kaldari> | 
  synchronized wmf-config/InitialiseSettings.php  'Bumping AFTv5 lottery percentage to 20% of en.wiki' | 
  [production] | 
            
  | 21:15 | 
  <LeslieCarr> | 
  powercycling unresponsive mw1116 | 
  [production] | 
            
  | 21:09 | 
  <maplebed> | 
  added new ms-be pmtpa hosts to DNS | 
  [production] | 
            
  | 20:03 | 
  <preilly> | 
  synchronized php-1.20wmf6/extensions/MobileFrontend  'fix cache issue with token' | 
  [production] | 
            
  | 19:14 | 
  <Jeff_Green> | 
  *somebody* purged binary logs on blondel | 
  [production] | 
            
  | 18:58 | 
  <preilly> | 
  synchronized php-1.20wmf6/extensions/MobileFrontend  'fix cache issue' | 
  [production] | 
            
  | 18:54 | 
  <mark> | 
  Upgraded pybal on all precise LVS servers | 
  [production] | 
            
  | 18:43 | 
  <mark> | 
  Built new pybal_1.03 package and inserted it into the precise-wikimedia APT repository | 
  [production] | 
            
  | 17:41 | 
  <mutante> | 
  powercycling db1048, mw1136, mw1046 | 
  [production] | 
            
  | 17:32 | 
  <midom> | 
  synchronized wmf-config/InitialiseSettings.php  'reenabling db40' | 
  [production] | 
            
  | 17:30 | 
  <mutante> | 
  more powercycling and upgrading: mw1036, mw1134, mw1043 .. | 
  [production] | 
            
  | 17:26 | 
  <mutante> | 
  powercycling db1009,db1010 | 
  [production] | 
            
  | 17:22 | 
  <mutante> | 
  powercycling db1027,db1028 | 
  [production] | 
            
  | 17:14 | 
  <mutante> | 
  powercycling db1013 | 
  [production] | 
            
  | 16:50 | 
  <mutante> | 
  powercycling downed db1015 | 
  [production] | 
            
  | 16:44 | 
  <mutante> | 
  powercycling and upgrading more mw10xx servers, 1017,1023,1025 ... | 
  [production] | 
            
  | 16:27 | 
  <mutante> | 
  mw1002, mw1007,mw1009,mw1011 - crashed,powercycling,dist-upgrading+kernel,reboot | 
  [production] | 
            
  | 16:18 | 
  <mutante> | 
  powercycling mw1002 | 
  [production] |