| 2018-05-09
      
      § | 
    
  | 08:17 | <demon@tin> | Started deploy [gerrit/gerrit@c421c91]: 2.14.7 -> 2.14.8 | [production] | 
            
  | 07:12 | <moritzm> | reimaging mw2152 to stretch (video scaler with a deprecated one-off partman recipe) | [production] | 
            
  | 07:10 | <moritzm> | reimaging mw2162 to stretch (last jessie job runner in codfw) | [production] | 
            
  | 07:05 | <moritzm> | reimaging mw1334, mw1335 (job runners) to stretch | [production] | 
            
  | 06:04 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2211.codfw.wmnet | [production] | 
            
  | 06:01 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2210.codfw.wmnet | [production] | 
            
  | 06:00 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2209.codfw.wmnet | [production] | 
            
  | 05:26 | <marostegui> | Stop MySQL on db2092 to do some clean ups | [production] | 
            
  | 05:19 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 01m 36s) | [production] | 
            
  | 05:19 | <marostegui> | Stop slave on db1116:s3 to do some gtid cleanups and tests | [production] | 
            
  | 04:02 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed May  9 04:02:56 UTC 2018 (duration 7m 26s) | [production] | 
            
  | 03:55 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.3) (duration: 16m 34s) | [production] | 
            
  | 02:57 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.2) (duration: 09m 06s) | [production] | 
            
  | 02:09 | <maxsem@tin> | Synchronized wmf-config: Preparation for T194230 (duration: 01m 15s) | [production] | 
            
  | 02:06 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings.php: Preparation for T194230 (duration: 01m 16s) | [production] | 
            
  | 02:02 | <maxsem@tin> | Synchronized php-1.32.0-wmf.2/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 17s) | [production] | 
            
  | 02:00 | <maxsem@tin> | Synchronized php-1.32.0-wmf.3/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 22s) | [production] | 
            
  | 01:36 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2211.codfw.wmnet | [production] | 
            
  | 01:36 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2210.codfw.wmnet | [production] | 
            
  | 01:36 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2209.codfw.wmnet | [production] | 
            
  | 01:36 | <XioNoX> | progressively push updated BGP_sanitize_in prefix-length-range to routers - T190317 | [production] | 
            
  | 01:33 | <mutante> | mw2209,mw2210,mw2211 - reinstall wtih stretch | [production] | 
            
  | 01:27 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2204.codfw.wmnet | [production] | 
            
  | 01:26 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2203.codfw.wmnet | [production] | 
            
  | 01:22 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2208.codfw.wmnet | [production] | 
            
  | 01:20 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2207.codfw.wmnet | [production] | 
            
  | 01:19 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2205.codfw.wmnet | [production] | 
            
  | 00:54 | <ejegg> | running refund queue consumer overtime | [production] | 
            
  | 00:35 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432022/ - noop in prod (duration: 01m 20s) | [production] | 
            
  
    | 2018-05-08
      
      § | 
    
  | 23:58 | <thcipriani@tin> | Synchronized php-1.32.0-wmf.2/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/LoggingHelper.php: SWAT: [[gerrit:431805|Do not try to access null message message key]] T194140 (duration: 01m 32s) | [production] | 
            
  | 23:42 | <XioNoX> | progressively push BGP_sanitize_in as-path too-many-hops to routers - T190317 | [production] | 
            
  | 23:25 | <ejegg> | updated CiviCRM from d9d412e496 to 4ff35ad4bb | [production] | 
            
  | 23:22 | <thcipriani@tin> | Synchronized wmf-config: SWAT: [[gerrit:431994|Add string and external-id types to Wikibase indexing]] T163642 T99899 (duration: 01m 26s) | [production] | 
            
  | 23:12 | <XioNoX> | lowering ospf metric of ulsfo-codfw to 390 | [production] | 
            
  | 22:53 | <awight@tin> | Finished deploy [ores/deploy@bf182e2]: Rollback ores1002 to master (duration: 00m 19s) | [production] | 
            
  | 22:52 | <awight@tin> | Started deploy [ores/deploy@bf182e2]: Rollback ores1002 to master | [production] | 
            
  | 22:49 | <awight@tin> | Finished deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 (duration: 01m 59s) | [production] | 
            
  | 22:47 | <awight@tin> | Started deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 | [production] | 
            
  | 22:35 | <XioNoX> | remove PREFERRED-TRANSIT Tele2-DTAG from esams/knams routers | [production] | 
            
  | 22:05 | <XioNoX> | progressively push updated BGP_sanitize_in bogon ASN filters to routers - T190317 | [production] | 
            
  | 21:59 | <twentyafterfour> | MediaWiki train for 1.32.0-wmf.3 group0 is complete. Will resume with group1 tomorrow, same bat time, same bat channel (refs T191049) | [production] | 
            
  | 21:45 | <twentyafterfour@tin> | rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.3 | [production] | 
            
  | 21:36 | <eileen> | civicrm revision changed from 81e54c850d to d9d412e496, config revision is 96fbded693 | [production] | 
            
  | 21:09 | <twentyafterfour@tin> | Finished scap: testwikis wikis to 1.32.0-wmf.3 (duration: 114m 21s) | [production] | 
            
  | 20:59 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2204.codfw.wmnet | [production] | 
            
  | 20:58 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2203.codfw.wmnet | [production] | 
            
  | 20:24 | <milimetric@tin> | Finished deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors (duration: 07m 07s) | [production] | 
            
  | 20:17 | <milimetric@tin> | Started deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors | [production] | 
            
  | 20:02 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2201.codfw.wmnet | [production] | 
            
  | 20:01 | <mutante> | mw2205,mw2206,mw2207 - reinstalling with stretch - mw2202 - wmf-auto-reimage failed: Timeout of 60 minutes reached waiting for reboot | [production] |