|
2021-02-15
§
|
| 13:33 |
<marostegui> |
Stop MySQL on db1093 - T273955 |
[production] |
| 13:19 |
<kharlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 13:06 |
<godog> |
swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 |
[production] |
| 13:05 |
<Lucas_WMDE> |
notice: stashbot had issues between 8:19 and 12:50, see for https://wm-bot.wmflabs.org/browser/index.php?start=02%2F15%2F2021&end=02%2F15%2F2021&display=%23wikimedia-operations for missed !log messages |
[production] |
| 13:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast4002.wikimedia.org with reason: REIMAGE |
[production] |
| 13:02 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:02 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on bast4002.wikimedia.org with reason: REIMAGE |
[production] |
| 12:58 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 12:58 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1162 (re)pooling @ 4%: Slowly pool db1162', diff saved to https://phabricator.wikimedia.org/P14343 and previous config saved to /var/cache/conftool/dbconfig/20210215-080435-root.json |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1162 (re)pooling @ 3%: Slowly pool db1162', diff saved to https://phabricator.wikimedia.org/P14342 and previous config saved to /var/cache/conftool/dbconfig/20210215-074932-root.json |
[production] |
| 07:42 |
<elukey@cumin1001> |
START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes - elukey@cumin1001 |
[production] |
| 07:33 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1010.eqiad.wmnet |
[production] |
| 07:28 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-tool1010.eqiad.wmnet |
[production] |
| 07:26 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1009.eqiad.wmnet |
[production] |
| 07:24 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-tool1009.eqiad.wmnet |
[production] |
| 07:22 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1008.eqiad.wmnet |
[production] |
| 07:20 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-tool1008.eqiad.wmnet |
[production] |
| 07:16 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet |
[production] |
| 07:14 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14341 and previous config saved to /var/cache/conftool/dbconfig/20210215-070206-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14340 and previous config saved to /var/cache/conftool/dbconfig/20210215-064628-marostegui.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1162 to dbctl - depooled T258361', diff saved to https://phabricator.wikimedia.org/P14339 and previous config saved to /var/cache/conftool/dbconfig/20210215-064001-marostegui.json |
[production] |
|
2021-02-14
§
|
| 13:13 |
<akosiaris> |
sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service' |
[production] |
| 13:10 |
<_joe_> |
restarted varnish-fe on cp5004 |
[production] |
| 13:09 |
<akosiaris> |
restart varnish-fe on cp5001 |
[production] |
| 09:27 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] (duration: 00m 06s) |
[production] |
| 09:27 |
<joal@deploy1001> |
Started deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] |
[production] |
| 09:27 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] (duration: 12m 52s) |
[production] |
| 09:14 |
<joal@deploy1001> |
Started deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] |
[production] |
|
2021-02-13
§
|
| 03:23 |
<ryankemper> |
Depooled `wdqs1006` to catch up on lag |
[production] |
| 03:23 |
<ryankemper> |
Restarted blazegraph on `wdqs1006` |
[production] |
| 01:30 |
<crusnov@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host mwdebug1002.eqiad.wmnet |
[production] |
| 01:00 |
<crusnov@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:55 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:38 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:31 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1281.eqiad.wmnet |
[production] |
| 00:31 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1282.eqiad.wmnet |
[production] |
| 00:31 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1283.eqiad.wmnet |
[production] |
| 00:31 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1284.eqiad.wmnet |
[production] |
| 00:30 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:30 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:26 |
<mutante> |
ganeti - attempting to recreate VM mwdebug1002 with cookbook that wsa previously deleted manually (T274689 T274023) |
[production] |
| 00:25 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet |
[production] |
| 00:08 |
<mutante> |
ganeti1011 - manually deleting VM mwdebug1002 - T274689 T274023 |
[production] |