|
2021-04-27
ยง
|
| 12:55 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts ms-be1019.eqiad.wmnet |
[production] |
| 12:46 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:682815|Revert "URGENT: Disable GlobalUsage" (T281242)]] (duration: 01m 08s) |
[production] |
| 12:44 |
<hashar> |
Restarted CI Jenkins for plugins upgrade |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15592 and previous config saved to /var/cache/conftool/dbconfig/20210427-122619-root.json |
[production] |
| 12:20 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.1/extensions/GlobalUsage: Backport: [[gerrit:682814|Avoid reading primary unless absolutely necessary (T281238)]] (duration: 01m 09s) |
[production] |
| 12:12 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.3/extensions/GlobalUsage: Backport: [[gerrit:682813|Avoid reading primary unless absolutely necessary (T281238)]] (duration: 01m 09s) |
[production] |
| 12:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15591 and previous config saved to /var/cache/conftool/dbconfig/20210427-121115-root.json |
[production] |
| 12:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on labstore1007.wikimedia.org with reason: T281045 |
[production] |
| 12:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on labstore1007.wikimedia.org with reason: T281045 |
[production] |
| 11:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15590 and previous config saved to /var/cache/conftool/dbconfig/20210427-115612-root.json |
[production] |
| 11:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15589 and previous config saved to /var/cache/conftool/dbconfig/20210427-114108-root.json |
[production] |
| 11:36 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) |
[production] |
| 11:30 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-mirror-maker |
[production] |
| 11:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove RW from commonswiki', diff saved to https://phabricator.wikimedia.org/P15588 and previous config saved to /var/cache/conftool/dbconfig/20210427-111016-marostegui.json |
[production] |
| 11:09 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Disable GlobalUsage (duration: 01m 08s) |
[production] |
| 10:40 |
<volans@cumin1001> |
dbctl commit (dc=all): 'S4 RO, outage', diff saved to https://phabricator.wikimedia.org/P15585 and previous config saved to /var/cache/conftool/dbconfig/20210427-104057-volans.json |
[production] |
| 10:18 |
<godog> |
swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 |
[production] |
| 10:06 |
<XioNoX> |
standardize management routers ACLs with Capirca - mr1-eqiad (last one) |
[production] |
| 10:01 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@759f82c]: Homer release v0.2.7 (duration: 02m 16s) |
[production] |
| 09:59 |
<ayounsi@deploy1002> |
Started deploy [homer/deploy@759f82c]: Homer release v0.2.7 |
[production] |
| 09:56 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@759f82c]: Homer release v0.2.7 (duration: 00m 22s) |
[production] |
| 09:56 |
<ayounsi@deploy1002> |
Started deploy [homer/deploy@759f82c]: Homer release v0.2.7 |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1157 for schema change', diff saved to https://phabricator.wikimedia.org/P15584 and previous config saved to /var/cache/conftool/dbconfig/20210427-093536-marostegui.json |
[production] |
| 09:35 |
<XioNoX> |
standardize management routers ACLs with Capirca - mr1-eqsin |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repool db1175', diff saved to https://phabricator.wikimedia.org/P15583 and previous config saved to /var/cache/conftool/dbconfig/20210427-093501-root.json |
[production] |
| 09:34 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1012.eqiad.wmnet |
[production] |
| 09:34 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1011.eqiad.wmnet |
[production] |
| 09:33 |
<moritzm> |
rolling restart of elastic in relforge* to pick up Java updates |
[production] |
| 09:32 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2010.codfw.wmnet |
[production] |
| 09:31 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2009.codfw.wmnet |
[production] |
| 09:31 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2008.codfw.wmnet |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repool db1175', diff saved to https://phabricator.wikimedia.org/P15582 and previous config saved to /var/cache/conftool/dbconfig/20210427-091957-root.json |
[production] |
| 09:19 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb1012.eqiad.wmnet |
[production] |
| 09:19 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb1011.eqiad.wmnet |
[production] |
| 09:17 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb2010.codfw.wmnet |
[production] |
| 09:16 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host rdb2010.codfw.wmnet |
[production] |
| 09:16 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb2010.codfw.wmnet |
[production] |
| 09:16 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb2009.codfw.wmnet |
[production] |
| 09:16 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host rdb2008.codfw.wmnet |
[production] |
| 09:16 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2007.codfw.wmnet |
[production] |
| 09:11 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) |
[production] |
| 09:11 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on rdb2010.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:09 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on rdb2009.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:07 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1012.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:06 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2010.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:05 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb1011.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:05 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on rdb1012.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repool db1175', diff saved to https://phabricator.wikimedia.org/P15581 and previous config saved to /var/cache/conftool/dbconfig/20210427-090454-root.json |
[production] |
| 09:04 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on rdb2009.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:04 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) |
[production] |