|
2022-10-11
§
|
| 06:35 |
<XioNoX> |
delete now unused VC ports on asw2-c4-eqiad - T313384 |
[production] |
| 06:34 |
<elukey> |
kill leftover process of bmansurov on an-airflow1002 to allow user cleanup via puppet |
[production] |
| 06:34 |
<elukey> |
kill leftover process of bmansurov on an-airflow1002 to allow user cleanup via puppet |
[analytics] |
| 03:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 03:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 03:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 03:06 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
|
2022-10-10
§
|
| 23:04 |
<wm-bot> |
<anticomposite> ./SULWatcher/manage.sh restert # Ping timeout, all bots disconnected |
[tools.stewardbots] |
| 23:04 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 23:03 |
<wm-bot> |
<anticomposite> ./stewardbots/StewardBot/manage.sh restart # Ping timeout not noticed by the bot |
[tools.stewardbots] |
| 21:26 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 21:19 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS bullseye |
[production] |
| 21:09 |
<wm-bot> |
<root> restart stashbot |
[tools.stashbot] |
| 21:08 |
<wm-bot> |
<root> restart wikibugs |
[tools.wikibugs] |
| 20:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:44 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:841151|Resize wordmark and tagline of Bengali Wikibooks (T319320)]] (duration: 07m 29s) |
[production] |
| 20:44 |
<wm-bot> |
<root> restart stashbot |
[tools.stashbot] |
| 20:43 |
<wm-bot> |
<root> restart wikibugs |
[tools.wikibugs] |
| 20:01 |
<wm-bot> |
<anticomposite> Restart all bots |
[tools.stewardbots] |
| 19:30 |
<taavi> |
rebooting all k8s worker nodes to clean up labstore1006/7 remains |
[tools] |
| 19:16 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:14 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:14 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns4004 |
[production] |
| 19:14 |
<robh@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dns4004.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 19:13 |
<robh@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host dns4004 |
[production] |
| 19:07 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host dns4004.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 18:39 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns4002.wikimedia.org |
[production] |
| 18:39 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:38 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4008.ulsfo.wmnet with OS bullseye |
[production] |
| 18:37 |
<wm-bot> |
<anticomposite> ./stewardbots/SULWatcher/manage.sh restart # All bots disconnected |
[tools.stewardbots] |
| 18:35 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:30 |
<robh@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts dns4002.wikimedia.org |
[production] |
| 18:21 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage |
[production] |
| 18:18 |
<robh@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage |
[production] |
| 17:20 |
<robh@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 17:17 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 17:17 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 17:09 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 17:01 |
<Rook> |
Bump pywikibot to 7.7.1 24d0e2223175024bd11465decbf0a1087c1cc2c3 T320432 |
[paws] |
| 17:01 |
<robh@cumin2002> |
END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 17:00 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti4008.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 16:58 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:56 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:56 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti4008 |
[production] |
| 16:55 |
<robh@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti4008 |
[production] |
| 16:51 |
<taavi> |
clean up labstore1006/7 mounts from k8s control nodes T320425 |
[tools] |