| 2011-10-11
      
      § | 
    
  | 14:43 | <py> | synchronized wmf-config/mc.php  'shuffling 1 more memecache instance for lucid upgrades' | [production] | 
            
  | 14:37 | <notpeter> | RoanKattouw also restarting mw68 | [production] | 
            
  | 14:36 | <RoanKattouw> | And mw73 too | [production] | 
            
  | 14:35 | <RoanKattouw> | Gracefulling Apache on mw69, mw70, mw72 | [production] | 
            
  | 14:34 | <RoanKattouw> | Gracefulling Apache on mw71, it's throwing weird notices | [production] | 
            
  | 14:31 | <py> | synchronized wmf-config/mc.php  'shuffling 2 more memecache instances for lucid upgrades' | [production] | 
            
  | 14:20 | <py> | synchronized wmf-config/mc.php  'shuffling 2 more memecache instances for lucid upgrades' | [production] | 
            
  | 14:19 | <catrope> | synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php  'Fix syntax error (!!)' | [production] | 
            
  | 14:15 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'null -> false' | [production] | 
            
  | 14:14 | <catrope> | synchronized wmf-config/CommonSettings.php  'null -> false' | [production] | 
            
  | 14:11 | <notpeter> | removing srv187-srv189, srv214-srv218, and srv250-srv257 from api pool for upgrades to lucid | [production] | 
            
  | 14:11 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Fix undefined var in ArticleFeedback config' | [production] | 
            
  | 14:10 | <catrope> | synchronized wmf-config/CommonSettings.php  'Fix undefined var in ArticleFeedback config' | [production] | 
            
  | 13:53 | <mark> | Backported lcms2 to lucid and installed it in apt repository lucid-wikimedia (updates RT #1696) | [production] | 
            
  | 08:23 | <p858snake|l> | Test wiki broken (500 error) since at least 1513AEST (bz is being silly for me, and so I don't forget) | [production] | 
            
  | 02:03 | <LocalisationUpdate> | failed | [production] | 
            
  | 01:12 | <mutante> | amssq51 - power up, dist-upgrade, reoot, squid clean | [production] | 
            
  | 01:07 | <mutante> | sq77,sq78 - dist-upgraded/kernel, rebooted | [production] | 
            
  | 00:30 | <mutante> | sq74,sq75,sq76 - dist-upgrade/kernel, reboot | [production] | 
            
  | 00:28 | <apergos> | running gzip on september banner log files in /a/static/uncompressed/udplogs on hume, form root screen session | [production] | 
            
  
    | 2011-10-10
      
      § | 
    
  | 23:24 | <mutante> | amssq33 - power up,dist-upgrade,squid clean,reboot | [production] | 
            
  | 22:40 | <mutante> | sq70 - power up,dist-upgrade,reboot | [production] | 
            
  | 22:28 | <mutante> | sq71,sq72,sq73 - dist-upgraded/kernel,rebooted | [production] | 
            
  | 21:11 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 21:14:28 UTC 2011 | [production] | 
            
  | 21:09 | <LocalisationUpdate> | failed | [production] | 
            
  | 21:03 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 21:05:53 UTC 2011 | [production] | 
            
  | 20:47 | <Guest48655> | sq66,sq68 - upgraded kernel/reboot | [production] | 
            
  | 20:34 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:36:46 UTC 2011 | [production] | 
            
  | 20:29 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:32:22 UTC 2011 | [production] | 
            
  | 20:17 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:20:06 UTC 2011 | [production] | 
            
  | 20:04 | <notpeter> | putting srv290-srv301 back into api pool | [production] | 
            
  | 19:58 | <reedy> | synchronized php-1.18/maintenance/wmf/clearMessageBlobs.php | [production] | 
            
  | 19:50 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 19:53:32 UTC 2011 | [production] | 
            
  | 19:25 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 19:28:09 UTC 2011 | [production] | 
            
  | 18:24 | <Jeff_Green> | spence gets a cleansing reboot | [production] | 
            
  | 17:48 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable ClickTracking on zhwiki as prequisite for [[bugzilla:31578|bug 31578]] Install AF' | [production] | 
            
  | 17:33 | <notpeter> | removing srv290-srv301 from api lb pool for upgrades to lucid | [production] | 
            
  | 17:09 | <RobH> | memcached started on srv256, should clear spence check | [production] | 
            
  | 16:55 | <notpeter> | putting mw62-mw74 into api pool | [production] | 
            
  | 16:16 | <RobH> | mw1102 voltage errors, poking around system, it may possibly reboot | [production] | 
            
  | 14:15 | <notpeter> | restarting nagios and ganglia on spence | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 02:25:33 UTC 2011 | [production] | 
            
  | 00:47 | <Jeff_Green> | srv290, srv292, srv294, srv298 power cycled, see RT #1685 for more info | [production] |