| 
      
        2017-06-06
      
      §
     | 
  
    
  | 23:59 | 
  <thcipriani@tin> | 
  Synchronized php-1.30.0-wmf.2/extensions/Flow/includes/Content/BoardContentHandler.php: SWAT: [[gerrit:357500|Revert "Throw when unserializing invalid Flow workflow metadata JSON"]] T166100 T156813 (duration: 00m 43s) | 
  [production] | 
            
  | 23:58 | 
  <thcipriani@tin> | 
  Synchronized php-1.30.0-wmf.4/extensions/Flow/includes/Content/BoardContentHandler.php: SWAT: [[gerrit:357501|Revert "Throw when unserializing invalid Flow workflow metadata JSON"]] T166100 T156813 (duration: 00m 45s) | 
  [production] | 
            
  | 23:56 | 
  <RainbowSprinkles> | 
  gerrit: back from reindexing | 
  [production] | 
            
  | 23:55 | 
  <RainbowSprinkles> | 
  gerrit: force stopping for a second to reindex accounts | 
  [production] | 
            
  | 23:17 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:357425|Disable page previews on wikispecies]] T166894 (duration: 00m 44s) | 
  [production] | 
            
  | 23:12 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:357426|Update ContentNamespaces for Commons Wiki]] T167077 (duration: 00m 46s) | 
  [production] | 
            
  | 21:57 | 
  <RainbowSprinkles> | 
  gerrit: restarting last time, didn't work like I wanted | 
  [production] | 
            
  | 21:53 | 
  <RainbowSprinkles> | 
  gerrit: restarting to test a config tweak | 
  [production] | 
            
  | 21:41 | 
  <mutante> | 
  contint1001 - graceful'ed Apache to deploy gerrit:351391 | 
  [production] | 
            
  | 21:19 | 
  <demon@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: unbreak mw.org pref page | 
  [production] | 
            
  | 20:21 | 
  <RainbowSprinkles> | 
  gerrit: Down for just a moment, finally doing point release on cobalt | 
  [production] | 
            
  | 19:57 | 
  <demon@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4 | 
  [production] | 
            
  | 19:45 | 
  <demon@tin> | 
  Finished scap: testwiki to wmf.4 + prepping l10n. again (x2) (duration: 20m 25s) | 
  [production] | 
            
  | 19:36 | 
  <mutante> | 
  cobalt - removed systemd unit file (that has issues with ulimit and isn't used yet) - ran "systemctl reset-failed" which cleared the "systemctl status" which made the Icinga check recover | 
  [production] | 
            
  | 19:24 | 
  <demon@tin> | 
  Started scap: testwiki to wmf.4 + prepping l10n. again (x2) | 
  [production] | 
            
  | 19:23 | 
  <demon@tin> | 
  scap failed: RuntimeError scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) (duration: 13m 32s) | 
  [production] | 
            
  | 19:23 | 
  <demon@tin> | 
  scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) | 
  [production] | 
            
  | 19:10 | 
  <demon@tin> | 
  Started scap: testwiki to wmf.4 + prepping l10n. again | 
  [production] | 
            
  | 19:08 | 
  <demon@tin> | 
  Synchronized README: No-op, just forcing co-master sync (duration: 01m 27s) | 
  [production] | 
            
  | 19:01 | 
  <demon@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: testwiki back to wmf.2 | 
  [production] | 
            
  | 18:55 | 
  <maxsem@tin> | 
  Finished scap: LoginNotify to testwiki - rebuild messages (duration: 38m 19s) | 
  [production] | 
            
  | 18:16 | 
  <maxsem@tin> | 
  Started scap: LoginNotify to testwiki - rebuild messages | 
  [production] | 
            
  | 18:15 | 
  <maxsem@tin> | 
  Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/357317/2 (duration: 00m 44s) | 
  [production] | 
            
  | 18:10 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/357317/2 (duration: 00m 44s) | 
  [production] | 
            
  | 18:03 | 
  <demon@tin> | 
  Finished scap: testwiki to wmf.3, prepping l10n cache (duration: 31m 58s) | 
  [production] | 
            
  | 17:31 | 
  <demon@tin> | 
  Started scap: testwiki to wmf.3, prepping l10n cache | 
  [production] | 
            
  | 16:53 | 
  <moritzm> | 
  installing wireshark security updates on trusty (jessie already fixed) | 
  [production] | 
            
  | 16:41 | 
  <bblack> | 
  rebooted lvs1007 (kernel update) | 
  [production] | 
            
  | 16:35 | 
  <bblack> | 
  rebooted lvs1007 (kernel update) | 
  [production] | 
            
  | 15:21 | 
  <otto@tin> | 
  Finished deploy [eventlogging/analytics@37233cd]: (no justification provided) (duration: 00m 04s) | 
  [production] | 
            
  | 15:21 | 
  <otto@tin> | 
  Started deploy [eventlogging/analytics@37233cd]: (no justification provided) | 
  [production] | 
            
  | 14:58 | 
  <moritzm> | 
  installing libsndfile security updates on trusty | 
  [production] | 
            
  | 14:01 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Restore db1094 original weight (duration: 00m 40s) | 
  [production] | 
            
  | 13:46 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase db1094 weight (duration: 00m 40s) | 
  [production] | 
            
  | 13:39 | 
  <elukey> | 
  shutdown analytics1033 and analytics1039 to replace their BBU - T166140 | 
  [production] | 
            
  | 13:33 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1094 with low weight (duration: 00m 40s) | 
  [production] | 
            
  | 12:58 | 
  <marostegui> | 
  Shutdown db1094 for maintenance - T166518 | 
  [production] | 
            
  | 12:58 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1094 for maintenance - T166518 (duration: 00m 39s) | 
  [production] | 
            
  | 12:51 | 
  <godog> | 
  upgrade scap to 3.5.8 - T127762 | 
  [production] | 
            
  | 12:41 | 
  <mobrovac@tin> | 
  Finished deploy [changeprop/deploy@e92dd66]: Bump src to bc8abf3 (duration: 01m 45s) | 
  [production] | 
            
  | 12:40 | 
  <mobrovac@tin> | 
  Started deploy [changeprop/deploy@e92dd66]: Bump src to bc8abf3 | 
  [production] | 
            
  | 12:16 | 
  <bblack> | 
  cp1049 - restaret varnish backend for mailbox lag | 
  [production] | 
            
  | 12:08 | 
  <gehel> | 
  kill stuck osm replication on maps1001 | 
  [production] | 
            
  | 11:28 | 
  <akosiaris@tin> | 
  Finished deploy [servermon/servermon@4a2288f]: (no justification provided) (duration: 00m 04s) | 
  [production] | 
            
  | 11:28 | 
  <akosiaris@tin> | 
  Started deploy [servermon/servermon@4a2288f]: (no justification provided) | 
  [production] | 
            
  | 11:16 | 
  <moritzm> | 
  uploaded ferm 2.3.2+wmf1 to apt.wikimedia.org/stretch-wikimedia (T166653) | 
  [production] | 
            
  | 11:02 | 
  <ladsgroup@tin> | 
  Synchronized wmf-config/Wikibase-production.php: Enabling writing in full entity id in testwikidatawiki (T165197) (duration: 00m 39s) | 
  [production] |