451-500 of 10000 results (83ms)
2024-05-03 ยง
11:44 <topranks> Removing connections from ssw1-a1-codfw and ssw1-a8-codfw to lsw1-a1-codfw T364097 [production]
11:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doh7002.wikimedia.org with reason: host reimage [production]
11:41 <marostegui@cumin1002> dbctl commit (dc=all): 'db1203 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61833 and previous config saved to /var/cache/conftool/dbconfig/20240503-114141-root.json [production]
11:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T361627)', diff saved to https://phabricator.wikimedia.org/P61832 and previous config saved to /var/cache/conftool/dbconfig/20240503-113924-marostegui.json [production]
11:38 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on doh7002.wikimedia.org with reason: host reimage [production]
11:27 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on durum7002.magru.wmnet with reason: host reimage [production]
11:26 <marostegui@cumin1002> dbctl commit (dc=all): 'db1203 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61831 and previous config saved to /var/cache/conftool/dbconfig/20240503-112635-root.json [production]
11:23 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on durum7002.magru.wmnet with reason: host reimage [production]
11:19 <taavi@cumin1002> END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) [production]
11:17 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM durum7001.magru.wmnet [production]
11:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1203.eqiad.wmnet with OS bookworm [production]
11:16 <taavi@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
11:16 <taavi@cumin1002> END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=93) [production]
11:15 <taavi@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
11:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2164 (T361627)', diff saved to https://phabricator.wikimedia.org/P61830 and previous config saved to /var/cache/conftool/dbconfig/20240503-111415-marostegui.json [production]
11:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
11:13 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
11:13 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
11:13 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
11:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163 (T361627)', diff saved to https://phabricator.wikimedia.org/P61829 and previous config saved to /var/cache/conftool/dbconfig/20240503-111337-marostegui.json [production]
11:12 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM durum7001.magru.wmnet [production]
11:11 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host doh7002.wikimedia.org with OS bookworm [production]
11:11 <marostegui@cumin1002> dbctl commit (dc=all): 'db1203 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61828 and previous config saved to /var/cache/conftool/dbconfig/20240503-111129-root.json [production]
11:11 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM doh7002.wikimedia.org - sukhe@cumin1002" [production]
11:10 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM doh7002.wikimedia.org - sukhe@cumin1002" [production]
11:09 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) doh7002.wikimedia.org on all recursors [production]
11:09 <sukhe@cumin1002> START - Cookbook sre.dns.wipe-cache doh7002.wikimedia.org on all recursors [production]
11:09 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:09 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM doh7002.wikimedia.org - sukhe@cumin1002" [production]
11:09 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM doh7001.wikimedia.org [production]
11:08 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM doh7002.wikimedia.org - sukhe@cumin1002" [production]
11:06 <sukhe@cumin1002> START - Cookbook sre.dns.netbox [production]
11:06 <sukhe@cumin1002> START - Cookbook sre.ganeti.makevm for new host doh7002.wikimedia.org [production]
11:05 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM doh7001.wikimedia.org [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ncredir7001.magru.wmnet [production]
11:00 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM ncredir7001.magru.wmnet [production]
10:58 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host durum7002.magru.wmnet with OS bookworm [production]
10:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240503-105824-marostegui.json [production]
10:56 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM netflow7001.magru.wmnet [production]
10:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage [production]
10:54 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM durum7002.magru.wmnet - sukhe@cumin1002" [production]
10:53 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM durum7002.magru.wmnet - sukhe@cumin1002" [production]
10:53 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum7002.magru.wmnet on all recursors [production]
10:53 <sukhe@cumin1002> START - Cookbook sre.dns.wipe-cache durum7002.magru.wmnet on all recursors [production]
10:53 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:53 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM durum7002.magru.wmnet - sukhe@cumin1002" [production]
10:52 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1203.eqiad.wmnet with reason: host reimage [production]
10:52 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM durum7002.magru.wmnet - sukhe@cumin1002" [production]
10:51 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM netflow7001.magru.wmnet [production]
10:50 <sukhe@cumin1002> START - Cookbook sre.dns.netbox [production]