851-900 of 10000 results (29ms)
2020-10-07 ยง
12:33 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
12:24 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
12:22 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
11:55 <_joe_> rolling restart of restbase due to running puppet with changed config-vars (a noop for the actual configuration) [production]
11:22 <Urbanecm> EU B&C window done [production]
11:22 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: f85bc3056f809910c0487fb0b0559b3de92b1992: Enable bot passwords at all fishbowl and private wikis (T258356) (duration: 00m 58s) [production]
11:15 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 57297362c0a22ecf16648b7be4a73c4cb80d53ef: Fix OAuthRateLimiter rate limit configuration (duration: 00m 59s) [production]
11:14 <urbanecm@deploy1001> sync-file aborted: 57297362c0a22ecf16648b7be4a73c4cb80d53ef: Fix OAuthRateLimiter rate limit configuration (duration: 00m 02s) [production]
11:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 6cdeea2c4c15780a641722157584f12febedab2a: Set CXMTThresholdForPublish to 95% for Vietnamese Wikipedia (T264161) (duration: 00m 59s) [production]
10:58 <marostegui> Set innodb_change_buffering = inserts on pc2009 T263443 [production]
09:53 <kormat@cumin1001> dbctl commit (dc=all): 'Remove db2119 from mw load groups T259831', diff saved to https://phabricator.wikimedia.org/P12945 and previous config saved to /var/cache/conftool/dbconfig/20201007-095355-kormat.json [production]
09:44 <kormat@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 100%: 75', diff saved to https://phabricator.wikimedia.org/P12944 and previous config saved to /var/cache/conftool/dbconfig/20201007-094412-kormat.json [production]
09:21 <moritzm> imported icu63 63.1-6+deb10u1~wmf1 to component/icu63 for stretch-wikimedia [production]
09:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1076 T264755 ', diff saved to https://phabricator.wikimedia.org/P12943 and previous config saved to /var/cache/conftool/dbconfig/20201007-090943-marostegui.json [production]
08:39 <kormat@cumin1001> dbctl commit (dc=all): 'db2138:3314 depooling: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12942 and previous config saved to /var/cache/conftool/dbconfig/20201007-083903-kormat.json [production]
08:38 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:38 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:32 <godog> roll-restart statsd-exporter across ms-be* after puppet run - T264588 [production]
08:09 <jayme> updated envoyproxy to 1.15.1-2 on all non mw and restbase hosts [production]
08:05 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:58 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2015 from dbctl T264700', diff saved to https://phabricator.wikimedia.org/P12941 and previous config saved to /var/cache/conftool/dbconfig/20201007-074951-marostegui.json [production]
07:14 <marostegui> Stop MySQL es2015 for decommissioning T264700 [production]
05:52 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
05:46 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
02:37 <eileen> civicrm revision changed from a30da7f92a to 39b4f954ed, config revision is 0ca9a3a055 [production]
01:00 <cdanis> repool esams; cr2-esams router upgrade complete [production]
00:43 <cdanis> T259621 cdanis@re1.cr2-esams> request chassis routing-engine master switch [production]
00:40 <cdanis> T259621 cdanis@re1.cr2-esams> request system reboot other-routing-engine [production]
00:36 <cdanis> T259621 cdanis@re1.cr2-esams> request system software add /var/tmp/junos-install-mx-x86-64-17.3R3-S8.1.tgz re0 no-validate [production]
00:26 <cdanis> T259621 cdanis@re0.cr2-esams> request chassis routing-engine master switch [production]
00:22 <cdanis> T259621 cdanis@re0.cr2-esams> request system reboot other-routing-engine [production]
00:15 <cdanis> T259621 cdanis@re0.cr2-esams> request system software add re1 no-validate /var/tmp/junos-install-mx-x86-64-17.3R3-S8.1.tgz [production]
00:01 <mutante> reinstalling testvm[345]001 to confirm OS installs work as normal after switching DHCP servers in POPs (T252526) [production]
2020-10-06 ยง
23:55 <mutante> ๐Ÿ–ง switched DHCP server for eqsin from install2003 to install5001 - homer deployed to cr*eqsin* (T252526) ๐Ÿ–ง [production]
23:53 <mutante> ๐Ÿ–ง switched DHCP server for ulsfo from install2003 to install4001 - homer deployed to cr*ulsfo* (T252526) ๐Ÿ–ง [production]
23:52 <mutante> ๐Ÿ–ง switched DHCP server for esams from install1003 to install3001 - homer deployed to cr*esams* (T252526) ๐Ÿ–ง [production]
23:43 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:11 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:07 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
22:32 <ryankemper> Restart of `wdqs-categories` done. WDQS deploy is complete [production]
21:57 <ryankemper> Restarting `wdqs-categories` across production instances one-at-a-time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'` [production]
21:57 <ryankemper> Restarting `wdqs-categories` across all test instances (not public facing): `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` [production]
21:56 <ryankemper> Restarting `wdqs-updater` across the fleet: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
21:55 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@e56a20e]: 0.3.51 (duration: 13m 09s) [production]
21:43 <ryankemper> All tests passing on canary `wdqs1003`, proceeding to rest of fleet [production]
21:42 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@e56a20e]: 0.3.51 [production]
21:14 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: gerrit:632535 (duration: 01m 00s) [production]
20:25 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:23 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]