1051-1100 of 10000 results (82ms)
2024-05-02 §
05:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136 (T361627)', diff saved to https://phabricator.wikimedia.org/P61656 and previous config saved to /var/cache/conftool/dbconfig/20240502-053654-marostegui.json [production]
05:27 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1181.eqiad.wmnet with OS bookworm [production]
05:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db2162 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61655 and previous config saved to /var/cache/conftool/dbconfig/20240502-052700-root.json [production]
05:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P61654 and previous config saved to /var/cache/conftool/dbconfig/20240502-052146-marostegui.json [production]
05:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
05:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
05:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS bookworm [production]
05:11 <marostegui@cumin1002> dbctl commit (dc=all): 'db2162 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61653 and previous config saved to /var/cache/conftool/dbconfig/20240502-051155-root.json [production]
05:06 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage [production]
05:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P61652 and previous config saved to /var/cache/conftool/dbconfig/20240502-050639-marostegui.json [production]
05:04 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: host reimage [production]
04:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage [production]
04:53 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: host reimage [production]
04:52 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1181.eqiad.wmnet with OS bookworm [production]
04:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136 (T361627)', diff saved to https://phabricator.wikimedia.org/P61651 and previous config saved to /var/cache/conftool/dbconfig/20240502-045131-marostegui.json [production]
04:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1181 T363892', diff saved to https://phabricator.wikimedia.org/P61650 and previous config saved to /var/cache/conftool/dbconfig/20240502-045017-root.json [production]
04:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db1236 to s7 primary and set section read-write T363892', diff saved to https://phabricator.wikimedia.org/P61649 and previous config saved to /var/cache/conftool/dbconfig/20240502-044848-marostegui.json [production]
04:48 <marostegui@cumin1002> dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - T363892', diff saved to https://phabricator.wikimedia.org/P61648 and previous config saved to /var/cache/conftool/dbconfig/20240502-044819-marostegui.json [production]
04:48 <marostegui> Starting s7 eqiad failover from db1181 to db1236 - T363892 [production]
04:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2136 (T361627)', diff saved to https://phabricator.wikimedia.org/P61647 and previous config saved to /var/cache/conftool/dbconfig/20240502-044020-marostegui.json [production]
04:40 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2136.codfw.wmnet with reason: Maintenance [production]
04:39 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2136.codfw.wmnet with reason: Maintenance [production]
04:35 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db2162.codfw.wmnet with OS bookworm [production]
04:34 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: Reimage [production]
04:34 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: Reimage [production]
04:34 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2162', diff saved to https://phabricator.wikimedia.org/P61646 and previous config saved to /var/cache/conftool/dbconfig/20240502-043403-root.json [production]
04:30 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363892 [production]
04:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db1236 with weight 0 T363892', diff saved to https://phabricator.wikimedia.org/P61645 and previous config saved to /var/cache/conftool/dbconfig/20240502-043019-marostegui.json [production]
04:30 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363892 [production]
04:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2099.codfw.wmnet with reason: Maintenance [production]
04:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2099.codfw.wmnet with reason: Maintenance [production]
04:28 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
04:27 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
2024-05-01 §
23:57 <eileen> civicrm upgraded from 3ac4043c to 80ae4543 [production]
21:37 <eileen> config revision changed from 36b287b6 to b772c8bc [production]
20:22 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] (duration: 19m 29s) [production]
20:10 <jdrewniak@deploy1002> jdlrobson and jdrewniak: Continuing with sync [production]
20:08 <jdrewniak@deploy1002> jdlrobson and jdrewniak: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:03 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:1025878|[Vector] Enable appearance menu and increased font-size on testwiki (T362147)]] [production]
19:40 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS bookworm [production]
19:40 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" [production]
19:39 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002" [production]
19:12 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage [production]
19:09 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage [production]
18:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
18:55 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
18:55 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236 (T361627)', diff saved to https://phabricator.wikimedia.org/P61644 and previous config saved to /var/cache/conftool/dbconfig/20240501-185521-marostegui.json [production]
18:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P61643 and previous config saved to /var/cache/conftool/dbconfig/20240501-184013-marostegui.json [production]
18:36 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm [production]
18:36 <sukhe@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS bookworm [production]