| 2013-05-31
      
      § | 
    
  | 14:40 | <reedy> | synchronized php-1.22wmf5/extensions/SecurePoll/ | [production] | 
            
  | 14:33 | <Reedy> | Created bv2013_edit tables on all wikis | [production] | 
            
  | 14:29 | <reedy> | synchronized php-1.22wmf4/extensions/SecurePoll/ | [production] | 
            
  | 14:28 | <reedy> | synchronized php-1.22wmf5/extensions/SecurePoll/ | [production] | 
            
  | 12:04 | <apergos> | uh... it would help if I looked on tin, the actual deploy host so nm. anyways texvc now working | [production] | 
            
  | 12:00 | <apergos> | ran scap-recompile to get texvc built for 1.22wmf5 and btw this had to be done from fenari, there's no dsh on terbium | [production] | 
            
  | 06:56 | <apergos> | ceph osds 50 and 132 on ms-fe1002 are logging 'slow requests', don't know how to restart specific osds in bobtail though | [production] | 
            
  | 06:42 | <Aaron|home> | Running a script on terbium to list out files that were updated (overwritten) but are not synced on all wikis (it found none on commons/enwiki) | [production] | 
            
  | 05:25 | <Tim> | on msfe1001-1004: disabled apache_status ganglia module | [production] | 
            
  | 04:17 | <Tim> | on nickel: restarting gmetad to see if that fixes ceph cluster reporting | [production] | 
            
  | 04:01 | <aaron> | synchronized wmf-config/filebackend.php  'Disabled ceph backends and switched to just swift.' | [production] | 
            
  | 02:31 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri May 31 02:31:40 UTC 2013 | [production] | 
            
  | 02:11 | <LocalisationUpdate> | completed (1.22wmf4) at Fri May 31 02:11:03 UTC 2013 | [production] | 
            
  | 02:06 | <LocalisationUpdate> | completed (1.22wmf5) at Fri May 31 02:06:06 UTC 2013 | [production] | 
            
  | 00:10 | <mwalker> | Updating payments deploy from dce20cbeb501a9945d97b2e64c59dbb4dd0a9620 to cf2d1639a9a8b2854d5302db2aba86199b4e4101 for RTBT attempted fix | [production] | 
            
  | 00:06 | <kaldari> | Finished syncing Wikimedia installation... : | [production] | 
            
  
    | 2013-05-30
      
      § | 
    
  | 23:58 | <LeslieCarr> | fixing packages/security upgrades on db45 | [production] | 
            
  | 23:56 | <kaldari> | Started syncing Wikimedia installation... : | [production] | 
            
  | 23:54 | <LeslieCarr> | mw1171 appears to have a bad hard drive | [production] | 
            
  | 22:07 | <mflaschen> | Finished syncing Wikimedia installation... : Deployment for E3 and E2 | [production] | 
            
  | 21:54 | <mflaschen> | Started syncing Wikimedia installation... : Deployment for E3 and E2 | [production] | 
            
  | 21:23 | <LeslieCarr> | powercycling mw16 | [production] | 
            
  | 20:34 | <cmjohnson1> | ms-be106 replacing disk slot 6 | [production] | 
            
  | 20:10 | <cmjohnson1> | es1001 replacing bad disk (slot 11) | [production] | 
            
  | 20:05 | <kaldari> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 19:54 | <LeslieCarr> | enabling routing-engine on asw-c1-eqiad | [production] | 
            
  | 19:53 | <LeslieCarr> | asw-c1-eqiad successfully replaced | [production] | 
            
  | 19:47 | <kaldari> | Started syncing Wikimedia installation... : | [production] | 
            
  | 19:21 | <LeslieCarr> | powering on new asw-c-eqiad switch -- possible disruption to row c connectivity (low risk) | [production] | 
            
  | 19:19 | <LeslieCarr> | replacing asw-c1-eqiad with new switch | [production] | 
            
  | 18:14 | <bblack> | updated vhtcpd deb to 0.0.6-1 on brewster, puppet will upgrade the varnishes | [production] | 
            
  | 16:18 | <andrewbogott_afk> | updated OpenStackManager on virt0 to git head. | [production] | 
            
  | 14:58 | <paravoid> | restarting ms-fe4 proxy | [production] | 
            
  | 14:41 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable SecurePoll on testwiki' | [production] | 
            
  | 13:20 | <RoanKattouw> | (it's not broken, never mind, I'm just incapable of reading things) | [production] | 
            
  | 13:20 | <RoanKattouw> | Synced afe912f7a335ae037b8d8b0ba9616dc55848a2f0 for VisualEditor updates (logmsgbot seems to be broken) | [production] | 
            
  | 13:20 | <hashar> | gallium: killed stalled Jenkins threads that were producing high I/O. | [production] | 
            
  | 13:15 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' | [production] | 
            
  | 13:14 | <catrope> | synchronized wmf-config/CommonSettings.php  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' | [production] | 
            
  | 13:14 | <catrope> | synchronized visualeditor.dblist  'afe912f7a335ae037b8d8b0ba9616dc55848a2f0' | [production] | 
            
  | 13:11 | <hashar> | gallium: blackholed some web crawler robot. | [production] | 
            
  | 12:01 | <Krinkle> | Gracefully reloading Zuul to deploy 'Ica1137438a1b9137f01276b4dfe49a82227de518' | [production] | 
            
  | 10:18 | <mark> | synchronized wmf-config/squid.php  'Add cp1037-1040 as text Varnish' | [production] | 
            
  | 02:37 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu May 30 02:37:00 UTC 2013 | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.22wmf4) at Thu May 30 02:14:49 UTC 2013 | [production] | 
            
  | 02:08 | <LocalisationUpdate> | completed (1.22wmf5) at Thu May 30 02:08:18 UTC 2013 | [production] |