|
2020-02-27
ยง
|
| 16:24 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:23 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:22 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:21 |
<moritzm> |
installing wget security updates on jessie |
[production] |
| 16:20 |
<vgutierrez> |
reimage lvs4005 with buster - T245984 |
[production] |
| 16:14 |
<bd808> |
Decommissioning tools-worker-10[21-40] |
[tools] |
| 16:12 |
<papaul> |
rebooting parse2009 to clear memory error |
[production] |
| 16:11 |
<Urbanecm> |
foreachwiki extensions/AbuseFilter/maintenance/fixOldLogEntries.php --verbose started (T228655) |
[production] |
| 16:10 |
<Urbanecm> |
mwscript extensions/AbuseFilter/maintenance/fixOldLogEntries.php --wiki=mediawikiwiki --verbose (T228655) |
[production] |
| 16:09 |
<vgutierrez> |
re-enable BGP in lvs4006 - T245984 |
[production] |
| 16:08 |
<addshore> |
begin warming wikidata term cache on db1126 for Q4-6 million T219123 |
[production] |
| 16:07 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q4M for the new term store for clients (was Q2M) + warm db1126 caches (T219123) cache bust (duration: 01m 04s) |
[production] |
| 16:05 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q4M for the new term store for clients (was Q2M) + warm db1126 caches (T219123) (duration: 01m 04s) |
[production] |
| 16:05 |
<moritzm> |
installing python3.7 security updates on Buster |
[production] |
| 16:03 |
<jeh> |
create 3 new VMs toolsbeta-elastic7-0[1,2,3] |
[toolsbeta] |
| 16:02 |
<bd808> |
Drained tools-worker-1021 |
[tools] |
| 16:02 |
<effie> |
disable puppet on thumbor* |
[production] |
| 16:00 |
<jeh> |
increase CloudVPS quota instance count for new elasticsearch servers |
[toolsbeta] |
| 15:59 |
<moritzm> |
installing e2fsck security updates on buster |
[production] |
| 15:56 |
<moritzm> |
installing python-django updates (packaged Debian version) |
[production] |
| 15:53 |
<James_F> |
Docker: Publish mediawiki-phan-testrun:0.1.3 for T226117 |
[releng] |
| 15:52 |
<moritzm> |
installing python-pysaml security updates |
[production] |
| 15:51 |
<bd808> |
Drained tools-worker-1022 |
[tools] |
| 15:44 |
<bd808> |
Drained tools-worker-1023 (there is no tools-worker-1024) |
[tools] |
| 15:39 |
<bd808> |
Drained tools-worker-1025 |
[tools] |
| 15:39 |
<bd808> |
Drained tools-worker-1026 |
[tools] |
| 15:37 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:35 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:32 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 04s) |
[production] |
| 15:31 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.21/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 05s) |
[production] |
| 15:29 |
<moritzm> |
restarting mw canaries to pick up curl update |
[production] |
| 15:23 |
<moritzm> |
installing curl security updates on stretch/buster |
[production] |
| 15:17 |
<vgutierrez> |
reimage lvs4006 with buster - T245984 |
[production] |
| 15:11 |
<bd808> |
Drained tools-worker-1027 |
[tools] |
| 15:09 |
<bd808> |
Drained tools-worker-1028 (there is no tools-worker-1029) |
[tools] |
| 15:07 |
<bd808> |
Drained tools-worker-1030 |
[tools] |
| 15:06 |
<bd808> |
Uncordoned tools-worker-10[16-20]. Was over optimistic about repacking legacy Kubernetes cluster into 15 instances. Will keep 20 for now. |
[tools] |
| 15:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 at 50% T245621', diff saved to https://phabricator.wikimedia.org/P10542 and previous config saved to /var/cache/conftool/dbconfig/20200227-150302-jynus.json |
[production] |
| 15:00 |
<bd808> |
Drained tools-worker-1031 |
[tools] |
| 14:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:54 |
<bd808> |
Hard reboot tools-worker-1016. Direct virsh console unresponsive. Stuck in shutdown since 2020-01-22? |
[tools] |
| 14:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:44 |
<bd808> |
Uncordoned tools-worker-1009.tools.eqiad.wmflabs |
[tools] |
| 14:41 |
<bd808> |
Drained tools-worker-1032 |
[tools] |
| 14:37 |
<bd808> |
Drained tools-worker-1033 |
[tools] |
| 14:35 |
<vgutierrez> |
reimage lvs4007 with buster - T245984 |
[production] |
| 14:35 |
<bd808> |
Drained tools-worker-1034 |
[tools] |
| 14:34 |
<bd808> |
Drained tools-worker-1035 |
[tools] |
| 14:33 |
<bd808> |
Drained tools-worker-1036 |
[tools] |
| 14:33 |
<bd808> |
Drained tools-worker-10{39,38,37} yesterday but did not !log |
[tools] |