7401-7450 of 10000 results (100ms)
2024-03-04 ยง
11:18 <cgoubert@cumin2002> START - Cookbook sre.hosts.reimage for host mw2320.codfw.wmnet with OS bullseye [production]
11:18 <cgoubert@cumin2002> START - Cookbook sre.hosts.reimage for host mw2316.codfw.wmnet with OS bullseye [production]
11:18 <cgoubert@cumin2002> START - Cookbook sre.hosts.reimage for host mw2315.codfw.wmnet with OS bullseye [production]
11:18 <cgoubert@cumin2002> START - Cookbook sre.hosts.reimage for host mw2314.codfw.wmnet with OS bullseye [production]
11:17 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2008-dev.codfw.wmnet with reason: host reimage [production]
11:14 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2007-dev.codfw.wmnet with reason: host reimage [production]
11:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1235 (T354015)', diff saved to https://phabricator.wikimedia.org/P58342 and previous config saved to /var/cache/conftool/dbconfig/20240304-111424-marostegui.json [production]
11:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance [production]
11:14 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance [production]
11:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234 (T354015)', diff saved to https://phabricator.wikimedia.org/P58341 and previous config saved to /var/cache/conftool/dbconfig/20240304-111401-marostegui.json [production]
11:12 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2008-dev.codfw.wmnet with reason: host reimage [production]
11:12 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2007-dev.codfw.wmnet with reason: host reimage [production]
11:08 <claime> Depooling mw2314.codfw.wmnet,mw2315.codfw.wmnet,mw2316.codfw.wmnet,mw2320.codfw.wmnet,mw2321.codfw.wmnet,mw2322.codfw.wmnet for move to k8s - T351074 [production]
10:59 <btullis@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [production]
10:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P58340 and previous config saved to /var/cache/conftool/dbconfig/20240304-105855-marostegui.json [production]
10:53 <taavi@cumin1002> START - Cookbook sre.hosts.reimage for host cloudnet2008-dev.codfw.wmnet with OS bookworm [production]
10:53 <taavi@cumin1002> START - Cookbook sre.hosts.reimage for host cloudnet2007-dev.codfw.wmnet with OS bookworm [production]
10:48 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on etherpad1003.eqiad.wmnet with reason: Shutdown and decommission old host [production]
10:48 <jelto@cumin1002> START - Cookbook sre.hosts.downtime for 10:00:00 on etherpad1003.eqiad.wmnet with reason: Shutdown and decommission old host [production]
10:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P58339 and previous config saved to /var/cache/conftool/dbconfig/20240304-104348-marostegui.json [production]
10:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1234 (T354015)', diff saved to https://phabricator.wikimedia.org/P58338 and previous config saved to /var/cache/conftool/dbconfig/20240304-102842-marostegui.json [production]
09:30 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
09:30 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
09:28 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:27 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
08:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 100%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58337 and previous config saved to /var/cache/conftool/dbconfig/20240304-080546-root.json [production]
07:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 75%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58336 and previous config saved to /var/cache/conftool/dbconfig/20240304-075041-root.json [production]
07:35 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 50%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58335 and previous config saved to /var/cache/conftool/dbconfig/20240304-073536-root.json [production]
07:32 <moritzm> installing tar security updates [production]
07:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 25%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58334 and previous config saved to /var/cache/conftool/dbconfig/20240304-072031-root.json [production]
07:08 <kart_> Updated cxserver to 2024-03-04-023843-production (T350773) [production]
07:08 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
07:07 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
07:06 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
07:05 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
07:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 10%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58333 and previous config saved to /var/cache/conftool/dbconfig/20240304-070526-root.json [production]
07:01 <kartik@deploy2002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
07:01 <kartik@deploy2002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
06:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 5%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58332 and previous config saved to /var/cache/conftool/dbconfig/20240304-065021-root.json [production]
06:35 <marostegui@cumin1002> dbctl commit (dc=all): 'db1186 (re)pooling @ 1%: After optimizing revision table', diff saved to https://phabricator.wikimedia.org/P58331 and previous config saved to /var/cache/conftool/dbconfig/20240304-063516-root.json [production]
06:27 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1186', diff saved to https://phabricator.wikimedia.org/P58330 and previous config saved to /var/cache/conftool/dbconfig/20240304-062703-root.json [production]
06:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2118.codfw.wmnet [production]
06:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2118.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002" [production]
06:19 <marostegui@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2118.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002" [production]
06:17 <marostegui@cumin1002> START - Cookbook sre.dns.netbox [production]
06:12 <marostegui@cumin1002> START - Cookbook sre.hosts.decommission for hosts db2118.codfw.wmnet [production]
03:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1234 (T354015)', diff saved to https://phabricator.wikimedia.org/P58329 and previous config saved to /var/cache/conftool/dbconfig/20240304-034333-marostegui.json [production]
03:43 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]
03:43 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance [production]