|
2025-04-30
§
|
| 14:27 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2002.codfw.wmnet |
[production] |
| 14:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75712 and previous config saved to /var/cache/conftool/dbconfig/20250430-142636-fceratto.json |
[production] |
| 14:21 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2002.codfw.wmnet |
[production] |
| 14:20 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2001.codfw.wmnet |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75711 and previous config saved to /var/cache/conftool/dbconfig/20250430-141845-fceratto.json |
[production] |
| 14:18 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183 (T392806)', diff saved to https://phabricator.wikimedia.org/P75710 and previous config saved to /var/cache/conftool/dbconfig/20250430-141819-fceratto.json |
[production] |
| 14:14 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2001.codfw.wmnet |
[production] |
| 14:14 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 14:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 14:11 |
<moritzm> |
failover Ganeti master in codfw to ganeti2021 |
[production] |
| 14:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet |
[production] |
| 14:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet |
[production] |
| 14:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2209.codfw.wmnet with reason: Maintenance |
[production] |
| 14:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P75709 and previous config saved to /var/cache/conftool/dbconfig/20250430-140312-fceratto.json |
[production] |
| 14:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet |
[production] |
| 14:01 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1223.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:00 |
<moritzm> |
installing libcap2 security updates |
[production] |
| 13:54 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 13:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2044.codfw.wmnet |
[production] |
| 13:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2044.codfw.wmnet |
[production] |
| 13:52 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1003.eqiad.wmnet |
[production] |
| 13:50 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P75708 and previous config saved to /var/cache/conftool/dbconfig/20250430-134805-fceratto.json |
[production] |
| 13:47 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2044.codfw.wmnet |
[production] |
| 13:46 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be1003.eqiad.wmnet |
[production] |
| 13:44 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1002.eqiad.wmnet |
[production] |
| 13:43 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 13:43 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 13:38 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be1002.eqiad.wmnet |
[production] |
| 13:36 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1001.eqiad.wmnet |
[production] |
| 13:36 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2044.codfw.wmnet |
[production] |
| 13:35 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore1004 — T392989, T390514 |
[production] |
| 13:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2043.codfw.wmnet |
[production] |
| 13:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2043.codfw.wmnet |
[production] |
| 13:33 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 13:33 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183 (T392806)', diff saved to https://phabricator.wikimedia.org/P75707 and previous config saved to /var/cache/conftool/dbconfig/20250430-133258-fceratto.json |
[production] |
| 13:32 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:31 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be1001.eqiad.wmnet |
[production] |
| 13:29 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2043.codfw.wmnet |
[production] |
| 13:29 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 13:27 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@deploy1003 ~ $ mwscript-k8s --comment=T392984 --follow -- namespaceDupes mswikisource --fix | tee T392984 |
[production] |
| 13:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1183 (T392806)', diff saved to https://phabricator.wikimedia.org/P75706 and previous config saved to /var/cache/conftool/dbconfig/20250430-132604-fceratto.json |
[production] |
| 13:25 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1183.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:25 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T392806)', diff saved to https://phabricator.wikimedia.org/P75705 and previous config saved to /var/cache/conftool/dbconfig/20250430-132539-fceratto.json |
[production] |
| 13:24 |
<jnuche@deploy1003> |
Installation of scap version "4.158.0" completed for 2 hosts |
[production] |
| 13:24 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2043.codfw.wmnet |
[production] |
| 13:23 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2042.codfw.wmnet |
[production] |