| 2017-09-28
      
      § | 
    
  | 18:54 | <thcipriani@tin> | Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: [[gerrit:381259|Revision::newFromTitle may return null]] T176882 (duration: 00m 50s) | [production] | 
            
  | 18:42 | <bblack> | cp4023 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 18:37 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:381251|Create rollbacker group at viwiktionary]] T176979 (duration: 00m 50s) | [production] | 
            
  | 18:26 | <thcipriani@tin> | Synchronized docroot/noc: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] PART II T104148 (duration: 00m 49s) | [production] | 
            
  | 18:25 | <thcipriani@tin> | Synchronized wmf-config: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] T104148 (duration: 00m 51s) | [production] | 
            
  | 17:57 | <arlolra> | Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832) | [production] | 
            
  | 17:52 | <mutante> | librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera | [production] | 
            
  | 17:46 | <arlolra@tin> | Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s) | [production] | 
            
  | 17:43 | <bblack> | cp4028 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:40 | <moritzm> | installing apache updates on cobalt | [production] | 
            
  | 17:32 | <bblack> | cp4022 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:30 | <arlolra@tin> | Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c | [production] | 
            
  | 17:25 | <marostegui> | Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua"  on terbium - T176985 | [production] | 
            
  | 17:20 | <bblack> | cp4027 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:08 | <bblack> | cp4021 - depool->reboot for bios update | [production] | 
            
  | 17:05 | <robh> | bios firmware uploaded and awaiting reboot for application on cp402[1235678] | [production] | 
            
  | 16:48 | <bd808> | Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978) | [production] | 
            
  | 16:26 | <robh> | cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891 | [production] | 
            
  | 16:09 | <robh> | cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish | [production] | 
            
  | 16:01 | <marostegui> | Global rename of CodeCat → Rua - T176985 | [production] | 
            
  | 15:22 | <hoo> | Ran scap pull on mwdebug1001 after experiments with T176844 | [production] | 
            
  | 14:54 | <dcausse> | restarting elastic on relforge servers to reinstall prod ltr plugin | [production] | 
            
  | 14:51 | <marostegui> | Stop MySQL on db1035 as server is going to be decommissioned - T176931 | [production] | 
            
  | 14:45 | <moritzm> | installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily) | [production] | 
            
  | 14:42 | <marostegui> | Remove db1035 from tendril - T176931 | [production] | 
            
  | 14:31 | <moritzm> | upgrading mw1276 (API canary) to HHVM 3.18.5 | [production] | 
            
  | 14:28 | <moritzm> | upgrading mwdebug* to HHVM 3.18.5 | [production] | 
            
  | 13:56 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s) | [production] | 
            
  | 13:55 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s) | [production] | 
            
  | 13:40 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 13:38 | <zfilipin@tin> | Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: [[gerrit:381210|ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017)]] (duration: 00m 50s) | [production] | 
            
  | 13:37 | <moritzm> | upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5 | [production] | 
            
  | 13:35 | <moritzm> | uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org | [production] | 
            
  | 13:26 | <marostegui> | Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509 | [production] | 
            
  | 13:13 | <ema> | bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882 | [production] | 
            
  | 13:12 | <marostegui> | Drop  redundant indexes from pagelinks and templatelinks on s4 - T174509 | [production] | 
            
  | 13:10 | <moritzm> | upgrading mw1261 to HHVM 3.18.5 | [production] | 
            
  | 13:08 | <ema> | bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882 | [production] | 
            
  | 13:02 | <marostegui> | Optimize template links and pagelinks on db2053 and db2046 - T174509 | [production] | 
            
  | 13:00 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s) | [production] | 
            
  | 12:48 | <dcausse> | restarting elastic on relforge servers to test a new version of the ltr plugin | [production] | 
            
  | 12:27 | <ema> | reboot cp402[2-6] to disable numa isolation | [production] | 
            
  | 10:30 | <gehel> | restart elasticsearch on relforge to validate new logging config - T175242 | [production] | 
            
  | 10:09 | <elukey> | incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally) | [production] | 
            
  | 10:00 | <elukey@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet | [production] | 
            
  | 09:59 | <elukey> | added new mediawiki jobrunner - mw1308 | [production] | 
            
  | 09:52 | <ema> | upgrade pybal to 1.14.0 on lvs1009-10 | [production] | 
            
  | 09:49 | <aaron@tin> | Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s) | [production] | 
            
  | 09:42 | <ema> | upgrade pybal to 1.14.0 on lvs400[13] | [production] | 
            
  | 08:00 | <elukey@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet | [production] |