| 2021-01-12
      
      § | 
    
  | 14:14 | <jmm@cumin2001> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) | [production] | 
            
  | 14:05 | <liw@deploy1001> | rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.26 | [production] | 
            
  | 14:00 | <jmm@cumin2001> | START - Cookbook sre.hosts.reboot-single | [production] | 
            
  | 13:55 | <moritzm> | draining ganeti3003 for eventual reboot | [production] | 
            
  | 13:53 | <moritzm> | failover ganeti master in esams to ganeti3002 | [production] | 
            
  | 13:44 | <jmm@cumin2001> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) | [production] | 
            
  | 13:39 | <jmm@cumin2001> | START - Cookbook sre.hosts.reboot-single | [production] | 
            
  | 13:33 | <moritzm> | draining ganeti3002 for eventual reboot | [production] | 
            
  | 12:19 | <jmm@cumin2001> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) | [production] | 
            
  | 12:12 | <jmm@cumin2001> | START - Cookbook sre.hosts.reboot-single | [production] | 
            
  | 12:08 | <moritzm> | draining ganeti3001 for eventual reboot | [production] | 
            
  | 11:22 | <moritzm> | installing edk2 security updates | [production] | 
            
  | 10:51 | <godog> | swift eqiad-prod: add weight to ms-be106[0-3] - T268435 | [production] | 
            
  | 10:28 | <aborrero@cumin1001> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cloudnet1004.eqiad.wmnet with reason: T271058 | [production] | 
            
  | 10:28 | <aborrero@cumin1001> | START - Cookbook sre.hosts.downtime for 1:00:00 on cloudnet1004.eqiad.wmnet with reason: T271058 | [production] | 
            
  | 10:26 | <moritzm> | installing systemd bugfix update from Buster 10.7 point release | [production] | 
            
  | 10:15 | <liw@deploy1001> | Finished scap: testwikis wikis to 1.36.0-wmf.26 (duration: 67m 18s) | [production] | 
            
  | 10:13 | <marostegui> | Restart mysql on db1138 to pick up new config T271427 T271106 | [production] | 
            
  | 10:12 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool db1138', diff saved to https://phabricator.wikimedia.org/P13736 and previous config saved to /var/cache/conftool/dbconfig/20210112-101211-marostegui.json | [production] | 
            
  | 09:28 | <liw@deploy1001> | Started scap: testwikis wikis to 1.36.0-wmf.26 | [production] | 
            
  | 09:05 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13732 and previous config saved to /var/cache/conftool/dbconfig/20210112-090533-root.json | [production] | 
            
  | 08:50 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13731 and previous config saved to /var/cache/conftool/dbconfig/20210112-085030-root.json | [production] | 
            
  | 08:49 | <reedy@deploy1001> | Synchronized wmf-config/InitialiseSettings-labs.php: T271755 (duration: 00m 57s) | [production] | 
            
  | 08:47 | <liw> | 1.36.0-wmf.26 was branched at e6ad9ab7713ee33c30cd7c17762737870dc8fd08 for T267419 | [production] | 
            
  | 08:40 | <marostegui> | Sanitize bclwiktionary diqwiktionary niawiki niawiktionary diqwiktionary on db1124  db2094 db11154 T270280 T270276 T270414 T270410 T271261 | [production] | 
            
  | 08:35 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13730 and previous config saved to /var/cache/conftool/dbconfig/20210112-083526-root.json | [production] | 
            
  | 08:30 | <moritzm> | installing remaining curl security updates on stretch | [production] | 
            
  | 08:21 | <marostegui> | Deploy schema change on s3 eqiad master - T270187 | [production] | 
            
  | 08:20 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13729 and previous config saved to /var/cache/conftool/dbconfig/20210112-082023-root.json | [production] | 
            
  | 08:04 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P13728 and previous config saved to /var/cache/conftool/dbconfig/20210112-080419-marostegui.json | [production] | 
            
  | 07:00 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1075 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13727 and previous config saved to /var/cache/conftool/dbconfig/20210112-070051-root.json | [production] | 
            
  | 06:53 | <XioNoX> | push CR655445, only configure vlans relevant to a switch | [production] | 
            
  | 06:45 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1075 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13726 and previous config saved to /var/cache/conftool/dbconfig/20210112-064548-root.json | [production] | 
            
  | 06:30 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1075 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13725 and previous config saved to /var/cache/conftool/dbconfig/20210112-063044-root.json | [production] | 
            
  | 06:30 | <jhuneidi@deploy1001> | Pruned MediaWiki: 1.36.0-wmf.21 (duration: 03m 21s) | [production] | 
            
  | 06:16 | <marostegui> | Stop mysql on db1079 to clone db1155:3317 T268742 | [production] | 
            
  | 06:15 | <marostegui@cumin1001> | dbctl commit (dc=all): 'db1075 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13724 and previous config saved to /var/cache/conftool/dbconfig/20210112-061541-root.json | [production] | 
            
  | 06:05 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P13723 and previous config saved to /var/cache/conftool/dbconfig/20210112-060557-marostegui.json | [production] | 
            
  | 05:59 | <marostegui@cumin1001> | dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P13722 and previous config saved to /var/cache/conftool/dbconfig/20210112-055953-marostegui.json | [production] | 
            
  
    | 2021-01-11
      
      § | 
    
  | 22:16 | <eileen> | process-control config revision is f08249ecf9 eoy jobs disabled | [production] | 
            
  | 22:12 | <eileen> | civicrm revision changed from 2df572bdcd to f417a510a5, config revision is f08249ecf9 | [production] | 
            
  | 21:58 | <Amir1> | deleting watchlist enteries of Fawikibot in fawiki (1.1M rows) | [production] | 
            
  | 21:20 | <mutante> | docker images - [deneb:/srv/images/production-images] $ sudo -i build-production-images | [production] | 
            
  | 21:02 | <bblack> | dns4002 - upgrade gdnsd to 3.5.0 package | [production] | 
            
  | 20:47 | <bblack> | authdns2001 - upgrade gdnsd to 3.5.0 package | [production] | 
            
  | 19:53 | <otto@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Migrate UniversalLanguageSelector to Event Platform - T268517 (duration: 00m 57s) | [production] | 
            
  | 19:43 | <Amir1> | end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T270417 T270413 T270279) | [production] | 
            
  | 19:14 | <Amir1> | start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T270417 T270413 T270279) | [production] | 
            
  | 18:48 | <dpifke@deploy1001> | Finished deploy [performance/arc-lamp@6bbac6d]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/650277 (duration: 00m 04s) | [production] | 
            
  | 18:48 | <dpifke@deploy1001> | Started deploy [performance/arc-lamp@6bbac6d]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/650277 | [production] |