|
2025-01-27
§
|
| 07:46 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2004.wikimedia.org |
[production] |
| 07:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host idp2004.wikimedia.org |
[production] |
| 07:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72439 and previous config saved to /var/cache/conftool/dbconfig/20250127-074030-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1159.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:35 |
<moritzm> |
installing tomcat10 security updates |
[production] |
| 07:25 |
<slyngshede@dns1004> |
END - running authdns-update |
[production] |
| 07:23 |
<slyngshede@dns1004> |
START - running authdns-update |
[production] |
| 06:54 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2204.codfw.wmnet with reason: Maintenance |
[production] |
| 06:24 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:23 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Index rebuild + upgrade |
[production] |
|
2025-01-26
§
|
| 22:07 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 |
[tools] |
| 22:04 |
<andrewbogott> |
restarting Node tools-k8s-worker-nfs-44 , too many D processes |
[tools] |
| 22:03 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 |
[tools] |
| 22:02 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44 |
[tools] |
| 22:02 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44 |
[tools] |
| 21:40 |
<mvernon@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: needs index repair |
[production] |
| 21:39 |
<mvernon@cumin2002> |
dbctl commit (dc=all): 'Depool db2182', diff saved to https://phabricator.wikimedia.org/P72438 and previous config saved to /var/cache/conftool/dbconfig/20250126-213912-mvernon.json |
[production] |
| 20:30 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Index rebuild + upgrade |
[production] |
| 17:43 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 17:43 |
<andrew@cumin1002> |
END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:43 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:37 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:18 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:18 |
<andrew@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:18 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 17:18 |
<andrew@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 16:44 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 16:44 |
<andrew@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 15:28 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 15:23 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:23 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:23 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:23 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:22 |
<andrew@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1013.eqiad.wmnet with OS bullseye |
[production] |
| 15:22 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:22 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:22 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:22 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:21 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 15:21 |
<andrew@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet'] |
[production] |
| 08:38 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud |
[tools] |
| 08:37 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud |
[tools] |
| 08:37 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster |
[tools] |
| 08:37 |
<taavi@cloudcumin1001> |
Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 08:27 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T384790) |
[tools] |
| 08:26 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster |
[tools] |
| 08:26 |
<taavi@cloudcumin1001> |
Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 08:16 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T384790) |
[tools] |
| 08:16 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster |
[tools] |
| 08:16 |
<taavi@cloudcumin1001> |
Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |