|
2024-05-02
§
|
| 04:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2136 (T361627)', diff saved to https://phabricator.wikimedia.org/P61651 and previous config saved to /var/cache/conftool/dbconfig/20240502-045131-marostegui.json |
[production] |
| 04:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1181 T363892', diff saved to https://phabricator.wikimedia.org/P61650 and previous config saved to /var/cache/conftool/dbconfig/20240502-045017-root.json |
[production] |
| 04:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write T363892', diff saved to https://phabricator.wikimedia.org/P61649 and previous config saved to /var/cache/conftool/dbconfig/20240502-044848-marostegui.json |
[production] |
| 04:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - T363892', diff saved to https://phabricator.wikimedia.org/P61648 and previous config saved to /var/cache/conftool/dbconfig/20240502-044819-marostegui.json |
[production] |
| 04:48 |
<marostegui> |
Starting s7 eqiad failover from db1181 to db1236 - T363892 |
[production] |
| 04:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2136 (T361627)', diff saved to https://phabricator.wikimedia.org/P61647 and previous config saved to /var/cache/conftool/dbconfig/20240502-044020-marostegui.json |
[production] |
| 04:40 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2136.codfw.wmnet with reason: Maintenance |
[production] |
| 04:39 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2136.codfw.wmnet with reason: Maintenance |
[production] |
| 04:35 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db2162.codfw.wmnet with OS bookworm |
[production] |
| 04:34 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: Reimage |
[production] |
| 04:34 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: Reimage |
[production] |
| 04:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2162', diff saved to https://phabricator.wikimedia.org/P61646 and previous config saved to /var/cache/conftool/dbconfig/20240502-043403-root.json |
[production] |
| 04:30 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363892 |
[production] |
| 04:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1236 with weight 0 T363892', diff saved to https://phabricator.wikimedia.org/P61645 and previous config saved to /var/cache/conftool/dbconfig/20240502-043019-marostegui.json |
[production] |
| 04:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363892 |
[production] |
| 04:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2099.codfw.wmnet with reason: Maintenance |
[production] |
| 04:29 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2099.codfw.wmnet with reason: Maintenance |
[production] |
| 04:28 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:27 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
|
2024-05-01
§
|
| 23:57 |
<eileen> |
civicrm upgraded from 3ac4043c to 80ae4543 |
[production] |
| 21:37 |
<eileen> |
config revision changed from 36b287b6 to b772c8bc |
[production] |
| 20:22 |
<jdrewniak@deploy1002> |
Finished scap: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] (duration: 19m 29s) |
[production] |
| 20:10 |
<jdrewniak@deploy1002> |
jdlrobson and jdrewniak: Continuing with sync |
[production] |
| 20:08 |
<jdrewniak@deploy1002> |
jdlrobson and jdrewniak: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:03 |
<jdrewniak@deploy1002> |
Started scap: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] |
[production] |
| 19:40 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS bookworm |
[production] |
| 19:40 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" |
[production] |
| 19:39 |
<sukhe@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" |
[production] |
| 19:12 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage |
[production] |
| 19:09 |
<sukhe@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage |
[production] |
| 18:55 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:55 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T361627)', diff saved to https://phabricator.wikimedia.org/P61644 and previous config saved to /var/cache/conftool/dbconfig/20240501-185521-marostegui.json |
[production] |
| 18:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P61643 and previous config saved to /var/cache/conftool/dbconfig/20240501-184013-marostegui.json |
[production] |
| 18:36 |
<sukhe@cumin1002> |
START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm |
[production] |
| 18:36 |
<sukhe@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS bookworm |
[production] |
| 18:36 |
<sukhe@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7002.magru.wmnet'] |
[production] |
| 18:35 |
<sukhe@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7002.magru.wmnet'] |
[production] |
| 18:35 |
<sukhe@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7002.magru.wmnet'] |
[production] |
| 18:35 |
<sukhe@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7002.magru.wmnet'] |
[production] |
| 18:28 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host cephosd1005.eqiad.wmnet with OS bookworm |
[production] |
| 18:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P61642 and previous config saved to /var/cache/conftool/dbconfig/20240501-182505-marostegui.json |
[production] |
| 18:16 |
<sukhe@cumin1002> |
START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm |
[production] |
| 18:15 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7001.wikimedia.org with OS bookworm |
[production] |
| 18:15 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" |
[production] |
| 18:14 |
<sukhe@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" |
[production] |
| 18:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T361627)', diff saved to https://phabricator.wikimedia.org/P61641 and previous config saved to /var/cache/conftool/dbconfig/20240501-180958-marostegui.json |
[production] |
| 18:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1236 (T361627)', diff saved to https://phabricator.wikimedia.org/P61640 and previous config saved to /var/cache/conftool/dbconfig/20240501-180645-marostegui.json |
[production] |
| 18:06 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:06 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |