|
2024-02-01
ยง
|
| 07:33 |
<slyngs> |
Failover debmonitor to new Bookworm host |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2110 (T355609)', diff saved to https://phabricator.wikimedia.org/P56030 and previous config saved to /var/cache/conftool/dbconfig/20240201-073053-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<moritzm> |
installing openjdk-11 security updates |
[production] |
| 07:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T355609)', diff saved to https://phabricator.wikimedia.org/P56029 and previous config saved to /var/cache/conftool/dbconfig/20240201-073031-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56028 and previous config saved to /var/cache/conftool/dbconfig/20240201-073015-root.json |
[production] |
| 07:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2104 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P56027 and previous config saved to /var/cache/conftool/dbconfig/20240201-072713-root.json |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2002.codfw.wmnet |
[production] |
| 07:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2104 T356374', diff saved to https://phabricator.wikimedia.org/P56026 and previous config saved to /var/cache/conftool/dbconfig/20240201-071934-root.json |
[production] |
| 07:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2107 to s2 primary and set section read-write T356374', diff saved to https://phabricator.wikimedia.org/P56025 and previous config saved to /var/cache/conftool/dbconfig/20240201-071831-marostegui.json |
[production] |
| 07:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T356374', diff saved to https://phabricator.wikimedia.org/P56024 and previous config saved to /var/cache/conftool/dbconfig/20240201-071807-marostegui.json |
[production] |
| 07:17 |
<marostegui> |
Starting s2 codfw failover from db2104 to db2107 - T356374 |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P56023 and previous config saved to /var/cache/conftool/dbconfig/20240201-071524-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56022 and previous config saved to /var/cache/conftool/dbconfig/20240201-071510-root.json |
[production] |
| 07:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet |
[production] |
| 07:13 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2002.codfw.wmnet |
[production] |
| 07:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet |
[production] |
| 07:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet |
[production] |
| 07:01 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T356374 |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db2107 with weight 0 T356374', diff saved to https://phabricator.wikimedia.org/P56021 and previous config saved to /var/cache/conftool/dbconfig/20240201-070057-marostegui.json |
[production] |
| 07:00 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s2 T356374 |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P56020 and previous config saved to /var/cache/conftool/dbconfig/20240201-070018-marostegui.json |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56019 and previous config saved to /var/cache/conftool/dbconfig/20240201-070005-root.json |
[production] |
| 06:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T355609)', diff saved to https://phabricator.wikimedia.org/P56018 and previous config saved to /var/cache/conftool/dbconfig/20240201-064511-marostegui.json |
[production] |
| 06:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56017 and previous config saved to /var/cache/conftool/dbconfig/20240201-064500-root.json |
[production] |
| 06:34 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:994879|ProductionServices.php: Promote pc2014 to pc1 master (T356068)]] (duration: 08m 53s) |
[production] |
| 06:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56016 and previous config saved to /var/cache/conftool/dbconfig/20240201-062955-root.json |
[production] |
| 06:28 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 06:27 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:994879|ProductionServices.php: Promote pc2014 to pc1 master (T356068)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:25 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:994879|ProductionServices.php: Promote pc2014 to pc1 master (T356068)]] |
[production] |
| 06:22 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:994720|Revert "db-production.php: Disable writes on es5"]] (duration: 08m 10s) |
[production] |
| 06:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356068 |
[production] |
| 06:21 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover pc1 T356068 |
[production] |
| 06:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2106 (T355609)', diff saved to https://phabricator.wikimedia.org/P56015 and previous config saved to /var/cache/conftool/dbconfig/20240201-062128-marostegui.json |
[production] |
| 06:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2106.codfw.wmnet with reason: Maintenance |
[production] |
| 06:21 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2106.codfw.wmnet with reason: Maintenance |
[production] |
| 06:15 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 06:15 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:994720|Revert "db-production.php: Disable writes on es5"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 1%: After switchover ', diff saved to https://phabricator.wikimedia.org/P56014 and previous config saved to /var/cache/conftool/dbconfig/20240201-061449-root.json |
[production] |
| 06:13 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:994720|Revert "db-production.php: Disable writes on es5"]] |
[production] |
| 06:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2024 T356235', diff saved to https://phabricator.wikimedia.org/P56013 and previous config saved to /var/cache/conftool/dbconfig/20240201-061041-root.json |
[production] |
| 06:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2023 to es5 primary T356235', diff saved to https://phabricator.wikimedia.org/P56012 and previous config saved to /var/cache/conftool/dbconfig/20240201-060853-root.json |
[production] |
| 06:07 |
<marostegui> |
Starting es4 codfw failover from es2024 to es2023 - T356235 |
[production] |
| 06:02 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:994611|db-production.php: Disable writes on es5 (T356235)]] (duration: 08m 19s) |
[production] |