|
2024-02-01
ยง
|
| 09:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119 (T355609)', diff saved to https://phabricator.wikimedia.org/P56044 and previous config saved to /var/cache/conftool/dbconfig/20240201-090607-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P56043 and previous config saved to /var/cache/conftool/dbconfig/20240201-085743-root.json |
[production] |
| 08:52 |
<hashar> |
Restarted primary Gerrit on gerrit1003 |
[production] |
| 08:44 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet |
[production] |
| 08:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P56042 and previous config saved to /var/cache/conftool/dbconfig/20240201-084238-root.json |
[production] |
| 08:42 |
<hashar> |
Restarting Gerrit replica on gerrit2002 |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2119 (T355609)', diff saved to https://phabricator.wikimedia.org/P56041 and previous config saved to /var/cache/conftool/dbconfig/20240201-084126-marostegui.json |
[production] |
| 08:41 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:41 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56040 and previous config saved to /var/cache/conftool/dbconfig/20240201-084104-marostegui.json |
[production] |
| 08:40 |
<vgutierrez@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet |
[production] |
| 08:33 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet |
[production] |
| 08:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P56039 and previous config saved to /var/cache/conftool/dbconfig/20240201-082733-root.json |
[production] |
| 08:26 |
<vgutierrez@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet |
[production] |
| 08:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P56038 and previous config saved to /var/cache/conftool/dbconfig/20240201-082558-marostegui.json |
[production] |
| 08:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2003.codfw.wmnet |
[production] |
| 08:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet |
[production] |
| 08:15 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet |
[production] |
| 08:13 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2003.codfw.wmnet |
[production] |
| 08:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P56036 and previous config saved to /var/cache/conftool/dbconfig/20240201-081228-root.json |
[production] |
| 08:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P56035 and previous config saved to /var/cache/conftool/dbconfig/20240201-081051-marostegui.json |
[production] |
| 07:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P56034 and previous config saved to /var/cache/conftool/dbconfig/20240201-075723-root.json |
[production] |
| 07:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56033 and previous config saved to /var/cache/conftool/dbconfig/20240201-075545-marostegui.json |
[production] |
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56032 and previous config saved to /var/cache/conftool/dbconfig/20240201-074520-root.json |
[production] |
| 07:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P56031 and previous config saved to /var/cache/conftool/dbconfig/20240201-074218-root.json |
[production] |
| 07:33 |
<slyngs> |
Failover debmonitor to new Bookworm host |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56030 and previous config saved to /var/cache/conftool/dbconfig/20240201-073053-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<moritzm> |
installing openjdk-11 security updates |
[production] |
| 07:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T355609)', diff saved to https://phabricator.wikimedia.org/P56029 and previous config saved to /var/cache/conftool/dbconfig/20240201-073031-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56028 and previous config saved to /var/cache/conftool/dbconfig/20240201-073015-root.json |
[production] |
| 07:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P56027 and previous config saved to /var/cache/conftool/dbconfig/20240201-072713-root.json |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2002.codfw.wmnet |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2104 T356374', diff saved to https://phabricator.wikimedia.org/P56026 and previous config saved to /var/cache/conftool/dbconfig/20240201-071934-root.json |
[production] |
| 07:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2107 to s2 primary and set section read-write T356374', diff saved to https://phabricator.wikimedia.org/P56025 and previous config saved to /var/cache/conftool/dbconfig/20240201-071831-marostegui.json |
[production] |
| 07:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T356374', diff saved to https://phabricator.wikimedia.org/P56024 and previous config saved to /var/cache/conftool/dbconfig/20240201-071807-marostegui.json |
[production] |
| 07:17 |
<marostegui> |
Starting s2 codfw failover from db2104 to db2107 - T356374 |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P56023 and previous config saved to /var/cache/conftool/dbconfig/20240201-071524-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56022 and previous config saved to /var/cache/conftool/dbconfig/20240201-071510-root.json |
[production] |
| 07:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet |
[production] |
| 07:13 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2002.codfw.wmnet |
[production] |
| 07:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet |
[production] |
| 07:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T356374 |
[production] |