|
2024-06-05
ยง
|
| 13:47 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.resource-report |
[production] |
| 13:46 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host prometheus3003.esams.wmnet |
[production] |
| 13:46 |
<elukey> |
factory reset for sretest1001 to test the new provision cookbook - T365372 |
[production] |
| 13:46 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host prometheus4002.ulsfo.wmnet |
[production] |
| 13:46 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ms-be2073.codfw.wmnet |
[production] |
| 13:46 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host prometheus5002.eqsin.wmnet |
[production] |
| 13:46 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host ms-be1074.eqiad.wmnet |
[production] |
| 13:45 |
<inflatador> |
bking@an-db1001 install acl pkg T363001 |
[production] |
| 13:43 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1073.eqiad.wmnet |
[production] |
| 13:43 |
<filippo@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet |
[production] |
| 13:43 |
<filippo@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7001.magru.wmnet |
[production] |
| 13:40 |
<daniel@deploy1002> |
daniel: Continuing with sync |
[production] |
| 13:39 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet |
[production] |
| 13:37 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet |
[production] |
| 13:37 |
<filippo@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host graphite1005.eqiad.wmnet |
[production] |
| 13:37 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host prometheus7001.magru.wmnet |
[production] |
| 13:37 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2072.codfw.wmnet |
[production] |
| 13:35 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host ms-be1073.eqiad.wmnet |
[production] |
| 13:35 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1072.eqiad.wmnet |
[production] |
| 13:34 |
<daniel@deploy1002> |
daniel: Backport for [[gerrit:1038688|Set LinterParseOnDerivedDataUpdate to false (T361013)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:34 |
<isaranto@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 13:30 |
<daniel@deploy1002> |
Started scap: Backport for [[gerrit:1038688|Set LinterParseOnDerivedDataUpdate to false (T361013)]] |
[production] |
| 13:29 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ms-be2072.codfw.wmnet |
[production] |
| 13:28 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2071.codfw.wmnet |
[production] |
| 13:27 |
<elukey> |
systemctl reset-failed prometheus-redis-exporter@6380.service redis-instance-tcp_6380.service on netbox[12]002 + apt-get purge of redis-server and prometheus-redis-exporter packages to clean up stale configs (no local redis is used) |
[production] |
| 13:27 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host ms-be1072.eqiad.wmnet |
[production] |
| 13:26 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1071.eqiad.wmnet |
[production] |
| 13:25 |
<dreamyjazz@deploy1002> |
Finished scap: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] (duration: 11m 39s) |
[production] |
| 13:25 |
<filippo@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host graphite1005.eqiad.wmnet |
[production] |
| 13:21 |
<fabfur> |
enable magru DC after applying IPIP encapsulation patches (T366466) |
[production] |
| 13:20 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ms-be2071.codfw.wmnet |
[production] |
| 13:19 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2070.codfw.wmnet |
[production] |
| 13:17 |
<dreamyjazz@deploy1002> |
dreamyjazz: Continuing with sync |
[production] |
| 13:17 |
<dreamyjazz@deploy1002> |
dreamyjazz: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2179 (T364299)', diff saved to https://phabricator.wikimedia.org/P64104 and previous config saved to /var/cache/conftool/dbconfig/20240605-131647-marostegui.json |
[production] |
| 13:16 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance |
[production] |
| 13:16 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance |
[production] |
| 13:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172 (T364299)', diff saved to https://phabricator.wikimedia.org/P64103 and previous config saved to /var/cache/conftool/dbconfig/20240605-131623-marostegui.json |
[production] |
| 13:14 |
<dreamyjazz@deploy1002> |
Started scap: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] |
[production] |
| 13:13 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ms-be2070.codfw.wmnet |
[production] |
| 13:13 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host ms-be1071.eqiad.wmnet |
[production] |
| 13:13 |
<dreamyjazz@deploy1002> |
Finished scap: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] (duration: 19m 13s) |
[production] |
| 13:10 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker |
[production] |
| 13:06 |
<fabfur> |
restarting pybal on lvs7001/lvs7003 to appy IPIP conf (T366466) |
[production] |
| 13:04 |
<dreamyjazz@deploy1002> |
dreamyjazz: Continuing with sync |
[production] |
| 13:03 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1070.eqiad.wmnet |
[production] |
| 13:02 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2069.codfw.wmnet |
[production] |
| 13:02 |
<dreamyjazz@deploy1002> |
dreamyjazz: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P64102 and previous config saved to /var/cache/conftool/dbconfig/20240605-130115-marostegui.json |
[production] |
| 12:56 |
<elukey@cumin1002> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker |
[production] |