| 2013-03-18
      
      § | 
    
  | 21:21 | <robh> | synchronized docroot | [production] | 
            
  | 21:18 | <LeslieCarr> | service apache2 start on the mw-eqiad group via dsh | [production] | 
            
  | 21:14 | <binasher> | started apache on all imagescalers | [production] | 
            
  | 21:00 | <robh> | gracefulled all apaches | [production] | 
            
  | 21:00 | <RobH> | troubleshooting erros in apache restart script | [production] | 
            
  | 20:58 | <robh> | gracefulled all apaches | [production] | 
            
  | 20:36 | <RobH> | authdns-update rt4694 | [production] | 
            
  | 20:20 | <RobH> | authdns-update | [production] | 
            
  | 20:04 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 20:00 | <reedy> | Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 | [production] | 
            
  | 19:44 | <^demon> | jetty freaked out again, forcing a gerrit restart | [production] | 
            
  | 19:32 | <binasher> | upgraded all coredb mariadb replicas to 5.5.30 | [production] | 
            
  | 19:25 | <reedy> | Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12 | [production] | 
            
  | 19:07 | <asher> | synchronized wmf-config/db-eqiad.php  'returning db1043 db1009' | [production] | 
            
  | 19:04 | <reedy> | synchronized docroot | [production] | 
            
  | 19:02 | <asher> | synchronized wmf-config/db-eqiad.php  'db1050 for special s1' | [production] | 
            
  | 18:56 | <asher> | synchronized wmf-config/db-eqiad.php  'pulling db1043, db1009 for upgrade' | [production] | 
            
  | 18:40 | <reedy> | synchronized php-1.21wmf12  'Initial sync of php-1.21wmf12' | [production] | 
            
  | 18:16 | <awight> | payments cluster updated from c19cc66 to fe4fb96b | [production] | 
            
  | 18:06 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 18:03 | <aaron> | synchronized wmf-config/PrivateSettings.php  'Removed old testing cruft' | [production] | 
            
  | 16:19 | <cmjohnson1> | removing/replacing disk 4 ms-be1004 | [production] | 
            
  | 16:01 | <hashar> | Hard restarted Zuul which was dead locked again :-( | [production] | 
            
  | 13:23 | <paravoid> | depooled, upgraded, restarted and pooled again all ms-fe[1-4] | [production] | 
            
  | 13:01 | <paravoid> | depooling ms-fe1 for new swift version (patched 1.7.4) | [production] | 
            
  | 11:27 | <apergos> | restarted both the varnishncsas on niobium, they were giant again | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013 | [production] | 
            
  
    | 2013-03-15
      
      § | 
    
  | 23:24 | <laner> | synchronized wmf-config/CommonSettings.php  'Disabling aft5' | [production] | 
            
  | 22:45 | <asher> | synchronized wmf-config/db-eqiad.php  'returning es1007,10 at lower weights' | [production] | 
            
  | 22:35 | <binasher> | rebooted es1007,1010 for kernel upgrades | [production] | 
            
  | 22:33 | <asher> | synchronized wmf-config/db-eqiad.php  'temporarily pulling es1007, es1010' | [production] | 
            
  | 22:23 | <Ryan_Lane> | hid domain dropdown for account creation and login on wikitech | [production] | 
            
  | 22:23 | <Ryan_Lane> | changed wikitech configuration to require email address on account creation | [production] | 
            
  | 22:22 | <Ryan_Lane> | deployed change 54093 to wikitech | [production] | 
            
  | 22:03 | <hashar> | Zuul : opened a bug about this week slowness, breakage:  https://bugzilla.wikimedia.org/show_bug.cgi?id=46176 | [production] | 
            
  | 21:51 | <aaron> | synchronized php-1.21wmf11/includes  'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc' | [production] | 
            
  | 21:36 | <binasher> | db59 -> mariadb 5.5.30 | [production] | 
            
  | 21:29 | <hashar> | restarted Zuul with the safe version ff79197 | [production] | 
            
  | 21:27 | <hashar> | downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events) | [production] | 
            
  | 21:25 | <binasher> | added new build of mariadb 5.5.30 to precise-wikimedia on brewster | [production] | 
            
  | 21:20 | <hashar> | hard restarted Zuul. It was locked down again :/ | [production] | 
            
  | 21:18 | <mutante> | installing package upgrades on gallium | [production] |