| 2018-05-09
      
      § | 
    
  | 01:36 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2209.codfw.wmnet | [production] | 
            
  | 01:36 | <XioNoX> | progressively push updated BGP_sanitize_in prefix-length-range to routers - T190317 | [production] | 
            
  | 01:33 | <mutante> | mw2209,mw2210,mw2211 - reinstall wtih stretch | [production] | 
            
  | 01:27 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2204.codfw.wmnet | [production] | 
            
  | 01:26 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2203.codfw.wmnet | [production] | 
            
  | 01:22 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2208.codfw.wmnet | [production] | 
            
  | 01:20 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2207.codfw.wmnet | [production] | 
            
  | 01:19 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2205.codfw.wmnet | [production] | 
            
  | 00:54 | <ejegg> | running refund queue consumer overtime | [production] | 
            
  | 00:35 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432022/ - noop in prod (duration: 01m 20s) | [production] | 
            
  
    | 2018-05-08
      
      § | 
    
  | 23:58 | <thcipriani@tin> | Synchronized php-1.32.0-wmf.2/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/LoggingHelper.php: SWAT: [[gerrit:431805|Do not try to access null message message key]] T194140 (duration: 01m 32s) | [production] | 
            
  | 23:42 | <XioNoX> | progressively push BGP_sanitize_in as-path too-many-hops to routers - T190317 | [production] | 
            
  | 23:25 | <ejegg> | updated CiviCRM from d9d412e496 to 4ff35ad4bb | [production] | 
            
  | 23:22 | <thcipriani@tin> | Synchronized wmf-config: SWAT: [[gerrit:431994|Add string and external-id types to Wikibase indexing]] T163642 T99899 (duration: 01m 26s) | [production] | 
            
  | 23:12 | <XioNoX> | lowering ospf metric of ulsfo-codfw to 390 | [production] | 
            
  | 22:53 | <awight@tin> | Finished deploy [ores/deploy@bf182e2]: Rollback ores1002 to master (duration: 00m 19s) | [production] | 
            
  | 22:52 | <awight@tin> | Started deploy [ores/deploy@bf182e2]: Rollback ores1002 to master | [production] | 
            
  | 22:49 | <awight@tin> | Finished deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 (duration: 01m 59s) | [production] | 
            
  | 22:47 | <awight@tin> | Started deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 | [production] | 
            
  | 22:35 | <XioNoX> | remove PREFERRED-TRANSIT Tele2-DTAG from esams/knams routers | [production] | 
            
  | 22:05 | <XioNoX> | progressively push updated BGP_sanitize_in bogon ASN filters to routers - T190317 | [production] | 
            
  | 21:59 | <twentyafterfour> | MediaWiki train for 1.32.0-wmf.3 group0 is complete. Will resume with group1 tomorrow, same bat time, same bat channel (refs T191049) | [production] | 
            
  | 21:45 | <twentyafterfour@tin> | rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.3 | [production] | 
            
  | 21:36 | <eileen> | civicrm revision changed from 81e54c850d to d9d412e496, config revision is 96fbded693 | [production] | 
            
  | 21:09 | <twentyafterfour@tin> | Finished scap: testwikis wikis to 1.32.0-wmf.3 (duration: 114m 21s) | [production] | 
            
  | 20:59 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2204.codfw.wmnet | [production] | 
            
  | 20:58 | <dzahn@neodymium> | conftool action : set/pooled=inactive; selector: name=mw2203.codfw.wmnet | [production] | 
            
  | 20:24 | <milimetric@tin> | Finished deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors (duration: 07m 07s) | [production] | 
            
  | 20:17 | <milimetric@tin> | Started deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors | [production] | 
            
  | 20:02 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2201.codfw.wmnet | [production] | 
            
  | 20:01 | <mutante> | mw2205,mw2206,mw2207 - reinstalling with stretch - mw2202 - wmf-auto-reimage failed: Timeout of 60 minutes reached waiting for reboot | [production] | 
            
  | 19:15 | <twentyafterfour@tin> | Started scap: testwikis wikis to 1.32.0-wmf.3 | [production] | 
            
  | 19:14 | <twentyafterfour> | testwikis to 1.32.0-wmf.3 - https://gerrit.wikimedia.org/r/#/c/431821/ refs T191049 | [production] | 
            
  | 19:11 | <twentyafterfour> | updated mediawiki changelog https://www.mediawiki.org/wiki/MediaWiki_1.32/wmf.3/Changelog refs T191049 | [production] | 
            
  | 18:55 | <mutante> | mw2202, mw2203, mw2204 - reinstall with stretch | [production] | 
            
  | 18:52 | <marostegui> | Manually fail disk #7 on db1073 to get it replaced | [production] | 
            
  | 18:22 | <mutante> | mwmaint1001 - rebooting | [production] | 
            
  | 18:18 | <twentyafterfour> | Branching MediaWiki master to wmf/1.32.0-wmf.3 refs T191049 | [production] | 
            
  | 18:08 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2252.codfw.wmnet | [production] | 
            
  | 18:07 | <andrew@tin> | Finished deploy [horizon/deploy@9245ca9]: rolling out member dashboard (duration: 03m 18s) | [production] | 
            
  | 18:06 | <dzahn@neodymium> | conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet | [production] | 
            
  | 18:03 | <andrew@tin> | Started deploy [horizon/deploy@9245ca9]: rolling out member dashboard | [production] | 
            
  | 17:00 | <bawolff> | Clear botpassword throttle for [[User:TaxonBot]] (T194160) | [production] | 
            
  | 16:45 | <thcipriani@tin> | Synchronized README: Testing Scap 3.8.1-1 (duration: 01m 02s) | [production] | 
            
  | 16:43 | <godog> | upload scap 3.8.1-1 - T127762 | [production] | 
            
  | 16:40 | <XioNoX> | re-pooling lvs2001 - T193677 | [production] | 
            
  | 16:36 | <mutante> | mwmaint1001 - reinstalling one more time after proxysql issues are resolved, PXE booting (T192092) | [production] | 
            
  | 16:27 | <mutante> | mw2251,mw2252,mw2201 - reinstall with stretch | [production] | 
            
  | 16:22 | <herron> | cleared low count edac counters on hosts mw2205 dbstore1002 db1051 elastic1029 T183177 | [production] | 
            
  | 16:19 | <urandom> | force (split) compaction of wikipedia_T_mobile__ng_lead.data, restbase1016 - T192689 | [production] |