|
2023-12-04
§
|
| 06:49 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:47 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2135,2160].codfw.wmnet,db[1119,1176,1217].eqiad.wmnet with reason: m5 master switch T352505 |
[production] |
| 06:46 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db[2135,2160].codfw.wmnet,db[1119,1176,1217].eqiad.wmnet with reason: m5 master switch T352505 |
[production] |
| 06:44 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:33 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:28 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:14 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:11 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:08 |
<kart_> |
Updated cxserver to 2023-12-04-055024-production (T270060, T350773, T352620) |
[production] |
| 06:06 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:05 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 06:03 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:02 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 05:59 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:58 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 04:43 |
<ryankemper> |
[WDQS] Clearing `BlazegraphFreeAllocatorsDecreasingRapidly` -> `ryankemper@wdqs1007:~$ sudo systemctl restart wdqs-blazegraph` |
[production] |
| 00:16 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet |
[production] |
| 00:09 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet |
[production] |
| 00:07 |
<andrewbogott> |
rebooting cloudcontrol1006 to recover from full disk error |
[admin] |
|
2023-12-02
§
|
| 13:42 |
<bd808> |
Restarted bot. Was missing from IRC channels (possibly all channels). Likely fallout from Kubernetes cluster issues yesterday. |
[tools.bridgebot] |
| 12:27 |
<taavi> |
powercycle cloudvirt1063 T352595 |
[admin] |
| 11:28 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |
| 11:28 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster |
[admin] |
| 11:18 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 11:15 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-22 |
[tools] |
| 11:06 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-13, tools-sgeweblight-10-20 |
[tools] |
| 11:02 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |
| 11:02 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster |
[admin] |
| 10:50 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 10:50 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |
| 10:50 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster |
[admin] |
| 10:39 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 01:51 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1078.eqiad.wmnet with OS bullseye |
[production] |
| 01:50 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1079.eqiad.wmnet with OS bullseye |
[production] |
| 01:50 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1077.eqiad.wmnet with OS bullseye |
[production] |
| 01:50 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1076.eqiad.wmnet with OS bullseye |
[production] |
| 00:30 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host ms-be1078.eqiad.wmnet with OS bullseye |
[production] |
| 00:30 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host ms-be1079.eqiad.wmnet with OS bullseye |
[production] |
| 00:30 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS bullseye |
[production] |
| 00:30 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host ms-be1076.eqiad.wmnet with OS bullseye |
[production] |
| 00:21 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |
| 00:21 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster |
[admin] |
| 00:18 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |
| 00:18 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster |
[admin] |
| 00:15 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] |
[production] |
| 00:15 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[admin] |