| 2017-09-29
      
      § | 
    
  | 18:06 | <chasemp> | remove grid-start-precise catchpoint check | [production] | 
            
  | 18:04 | <chasemp> | update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc | [production] | 
            
  | 17:00 | <ebernhardson> | kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index | [production] | 
            
  | 16:37 | <andrewbogott> | re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel | [production] | 
            
  | 15:52 | <andrewbogott> | re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel | [production] | 
            
  | 13:21 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042  - T174509 (duration: 00m 47s) | [production] | 
            
  | 13:18 | <Amir1> | starting a round of cleanup in ores_classification table in enwiki (T159753) | [production] | 
            
  | 13:09 | <elukey> | depool mw1265 (hhvm 3.18.5) - disk filled up | [production] | 
            
  | 12:43 | <bblack> | removed numa=isolate sys entries manually (runtime + future boots) from cp4021 | [production] | 
            
  | 11:53 | <bblack> | cp4021: reboot for numa mode switch | [production] | 
            
  | 11:44 | <elukey> | re-enable job runner daemons on mw130[8,9] | [production] | 
            
  | 11:15 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s) | [production] | 
            
  | 11:05 | <elukey> | precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation | [production] | 
            
  | 09:55 | <elukey@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet | [production] | 
            
  | 09:33 | <ema> | upgrade pybal to 1.14.0 on codfw primaries | [production] | 
            
  | 08:20 | <ema> | upgrade pybal to 1.14.0 on codfw secondaries | [production] | 
            
  | 05:53 | <ema> | cp4024 repooled T174891 | [production] | 
            
  | 05:20 | <ema> | powerup cp4024 T174891 | [production] | 
            
  | 00:26 | <ejegg> | updated fundraising tools from 69a96276fa8cd8f68aa1225ea48fff1a7c833101 to 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f | [production] | 
            
  | 00:16 | <ejegg> | updated CiviCRM 213886998cad62a37aab5b214290b8431ab41199 to 784756b4157517bf9639a67b8d0ddcb1db7c999d | [production] | 
            
  | 00:07 | <mutante> | releases1001 - starting Jenkins setup wizard with generated admin pass | [production] | 
            
  | 00:05 | <mutante> | releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up  T164030 | [production] | 
            
  
    | 2017-09-28
      
      § | 
    
  | 23:46 | <awight@tin> | Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s) | [production] | 
            
  | 23:45 | <awight@tin> | Started deploy [ores/deploy@42c5663]: Cause ORES service restart | [production] | 
            
  | 22:18 | <MaxSem> | Deployed fix for T176247 | [production] | 
            
  | 21:30 | <ejegg> | updated CiviCRM from 3f48cb03b4a216a34a61f9744a297de4fb99d8a6 to 213886998cad62a37aab5b214290b8431ab41199 | [production] | 
            
  | 21:23 | <XenoRyet> | Updated SmashPig  from dc6220391036c630c8ad14c0d894d33c1e8d8c2c to d056a131ccbbd4b505da4f99e04f5337fdd9c32e | [production] | 
            
  | 19:36 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1 | [production] | 
            
  | 19:34 | <legoktm> | mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798) | [production] | 
            
  | 19:31 | <bblack> | cp4026 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 19:28 | <legoktm> | mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua"; | [production] | 
            
  | 19:14 | <ejegg> | updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export | [production] | 
            
  | 19:14 | <robh> | cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far)  T174891 | [production] | 
            
  | 19:00 | <bblack> | cp4025 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 18:54 | <thcipriani@tin> | Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: [[gerrit:381259|Revision::newFromTitle may return null]] T176882 (duration: 00m 50s) | [production] | 
            
  | 18:42 | <bblack> | cp4023 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 18:37 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:381251|Create rollbacker group at viwiktionary]] T176979 (duration: 00m 50s) | [production] | 
            
  | 18:26 | <thcipriani@tin> | Synchronized docroot/noc: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] PART II T104148 (duration: 00m 49s) | [production] | 
            
  | 18:25 | <thcipriani@tin> | Synchronized wmf-config: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] T104148 (duration: 00m 51s) | [production] | 
            
  | 17:57 | <arlolra> | Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832) | [production] | 
            
  | 17:52 | <mutante> | librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera | [production] | 
            
  | 17:46 | <arlolra@tin> | Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s) | [production] | 
            
  | 17:43 | <bblack> | cp4028 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:40 | <moritzm> | installing apache updates on cobalt | [production] | 
            
  | 17:32 | <bblack> | cp4022 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:30 | <arlolra@tin> | Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c | [production] | 
            
  | 17:25 | <marostegui> | Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua"  on terbium - T176985 | [production] | 
            
  | 17:20 | <bblack> | cp4027 - depool->reboot for bios updates, isolcpus fix | [production] | 
            
  | 17:08 | <bblack> | cp4021 - depool->reboot for bios update | [production] | 
            
  | 17:05 | <robh> | bios firmware uploaded and awaiting reboot for application on cp402[1235678] | [production] |