| 2015-05-27
      
      § | 
    
  | 23:46 | <awight> | updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0 | [production] | 
            
  | 23:31 | <twentyafterfour> | Finished scap: scap, now with 10% less fail (duration: 22m 07s) | [production] | 
            
  | 23:26 | <awight> | payments rolled back to 858b87319daa3d66f62eb32e08cefc6b061748d1 | [production] | 
            
  | 23:24 | <awight> | updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0 | [production] | 
            
  | 23:09 | <twentyafterfour> | Started scap: scap, now with 10% less fail | [production] | 
            
  | 22:57 | <ori> | rebuilt wikiversions.cdb and synchronized wikiversions files: (no message) | [production] | 
            
  | 21:49 | <mutante> | restarted hhvm on mw1250,mw1254,mw1256 | [production] | 
            
  | 21:47 | <mutante> | restarted hhvm on mw1017,mw1243,mw1244 | [production] | 
            
  | 21:42 | <bblack> | restarting hhvm everywhere on 30s intervals between hosts | [production] | 
            
  | 21:10 | <twentyafterfour> | Synchronized php-1.26wmf8: Fix ConfirmEdit fatal Change-Id: I22353669a85391c3d9760a5253cac1263e895cf9 (duration: 01m 08s) | [production] | 
            
  | 20:47 | <twentyafterfour> | Purged l10n cache for 1.26wmf6 | [production] | 
            
  | 20:45 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf8 | [production] | 
            
  | 20:41 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf7 | [production] | 
            
  | 20:36 | <twentyafterfour> | Finished scap: testwiki to php-1.26wmf8 and rebuild l10n cache (duration: 67m 53s) | [production] | 
            
  | 19:28 | <twentyafterfour> | Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache | [production] | 
            
  | 19:18 | <twentyafterfour> | Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache | [production] | 
            
  | 18:12 | <moritzm> | Uploaded gridengine_6.2u5-4+wmf2 for precise-wikimedia to apt.wikimedia.org | [production] | 
            
  | 17:55 | <jynus> | Synchronized wmf-config/db-eqiad.php: repool pc1002 (duration: 00m 13s) | [production] | 
            
  | 17:42 | <paravoid> | rebooting asw-d2-eqiad | [production] | 
            
  | 17:41 | <ottomata> | initiating controlled shutdown of kafka broker analytics1018 in anticipation of switch reboot | [production] | 
            
  | 15:33 | <jynus> | Synchronized wmf-config/db-eqiad.php: depool pc1002 (duration: 00m 13s) | [production] | 
            
  | 15:02 | <cmjohnson1> | powering down cp1069 to relocate within the same rack | [production] | 
            
  | 14:47 | <cmjohnson1> | powering down cp1070 to relocate within the same rack | [production] | 
            
  | 13:30 | <hashar> | All Jenkins slaves are disconnected due to some ssh error. CI is down. | [production] | 
            
  | 13:27 | <hashar> | restarting Jenkins for java upgrade | [production] | 
            
  | 13:13 | <jynus> | Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 13s) | [production] | 
            
  | 11:16 | <akosiaris> | rebooting ganeti100{1..4} for bridge networking configuration | [production] | 
            
  | 09:59 | <paravoid> | powercycling ms-be1001; dead, console unresponsive | [production] | 
            
  | 06:35 | <springle> | clone dbstore2001 data to dbstore2002 | [production] | 
            
  | 05:48 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed May 27 05:47:25 UTC 2015 (duration 47m 24s) | [production] | 
            
  | 02:53 | <LocalisationUpdate> | completed (1.26wmf7) at 2015-05-27 02:52:25+00:00 | [production] | 
            
  | 02:48 | <l10nupdate> | Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 52s) | [production] | 
            
  | 02:29 | <LocalisationUpdate> | completed (1.26wmf6) at 2015-05-27 02:28:34+00:00 | [production] | 
            
  | 02:24 | <l10nupdate> | Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 45s) | [production] | 
            
  
    | 2015-05-26
      
      § | 
    
  | 18:21 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf7 | [production] | 
            
  | 17:13 | <krenair> | Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 15s) | [production] | 
            
  | 17:10 | <krenair> | Synchronized multiversion/MWMultiVersion.php: open cnwikimedia (duration: 00m 13s) | [production] | 
            
  | 16:27 | <krenair> | Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s) | [production] | 
            
  | 16:12 | <krenair> | rebuilt wikiversions.cdb and synchronized wikiversions files: add cnwikimedia | [production] | 
            
  | 16:08 | <krenair> | Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s) | [production] | 
            
  | 16:08 | <krenair> | Synchronized database lists: (no message) (duration: 00m 15s) | [production] | 
            
  | 16:07 | <krenair> | Synchronized w/static/images/project-logos/cnwikimedia.png: (no message) (duration: 00m 19s) | [production] | 
            
  | 15:52 | <jynus> | Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 14s) | [production] | 
            
  | 15:32 | <jynus> | Synchronized wmf-config/db-eqiad.php: repool db1063 (warm period) (duration: 00m 13s) | [production] | 
            
  | 15:24 | <krenair> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/213652/ (duration: 00m 15s) | [production] | 
            
  | 15:23 | <krenair> | Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/213257/ (duration: 00m 14s) | [production] | 
            
  | 14:54 | <bblack> | restarted ganglia-monitor on all cp* (many were obviously-broken, probably most recently from bad startup after the reboots last week) | [production] | 
            
  | 14:14 | <jynus> | Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 12s) | [production] | 
            
  | 08:24 | <jynus> | Synchronized wmf-config/db-eqiad.php: depool pc1001 (duration: 00m 13s) | [production] | 
            
  | 05:53 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue May 26 05:52:50 UTC 2015 (duration 52m 49s) | [production] |