1001-1050 of 10000 results (92ms)
2024-07-15 §
08:04 <volans@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s7 T369882 [production]
07:58 <slyngshede@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp2004.wikimedia.org - slyngshede@cumin1002" [production]
07:57 <slyngshede@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp2004.wikimedia.org - slyngshede@cumin1002" [production]
07:57 <slyngshede@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) idp2004.wikimedia.org on all recursors [production]
07:57 <slyngshede@cumin1002> START - Cookbook sre.dns.wipe-cache idp2004.wikimedia.org on all recursors [production]
07:57 <slyngshede@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:57 <slyngshede@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp2004.wikimedia.org - slyngshede@cumin1002" [production]
07:55 <slyngshede@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp2004.wikimedia.org - slyngshede@cumin1002" [production]
07:53 <slyngshede@cumin1002> START - Cookbook sre.dns.netbox [production]
07:53 <slyngshede@cumin1002> START - Cookbook sre.ganeti.makevm for new host idp2004.wikimedia.org [production]
07:36 <slyngshede@cumin1002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idp1004.wikimedia.org [production]
07:36 <slyngshede@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host idp1004.wikimedia.org with OS bookworm [production]
07:21 <slyngshede@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp1004.wikimedia.org with reason: host reimage [production]
07:17 <slyngshede@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on idp1004.wikimedia.org with reason: host reimage [production]
07:17 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1179.eqiad.wmnet with reason: T369855 [production]
07:17 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1179.eqiad.wmnet with reason: T369855 [production]
07:06 <slyngshede@cumin1002> START - Cookbook sre.hosts.reimage for host idp1004.wikimedia.org with OS bookworm [production]
07:05 <slyngshede@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp1004.wikimedia.org - slyngshede@cumin1002" [production]
07:04 <slyngshede@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM idp1004.wikimedia.org - slyngshede@cumin1002" [production]
07:04 <slyngshede@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) idp1004.wikimedia.org on all recursors [production]
07:04 <slyngshede@cumin1002> START - Cookbook sre.dns.wipe-cache idp1004.wikimedia.org on all recursors [production]
07:04 <slyngshede@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:04 <slyngshede@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp1004.wikimedia.org - slyngshede@cumin1002" [production]
07:03 <slyngshede@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM idp1004.wikimedia.org - slyngshede@cumin1002" [production]
07:00 <slyngshede@cumin1002> START - Cookbook sre.dns.netbox [production]
07:00 <slyngshede@cumin1002> START - Cookbook sre.ganeti.makevm for new host idp1004.wikimedia.org [production]
06:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repool db2136', diff saved to https://phabricator.wikimedia.org/P66473 and previous config saved to /var/cache/conftool/dbconfig/20240715-062216-root.json [production]
06:07 <dani@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
06:07 <dani@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
06:07 <dani@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
06:06 <dani@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
06:06 <dani@deploy1002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
06:06 <dani@deploy1002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
05:12 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host dbproxy2005.codfw.wmnet with OS bookworm [production]
04:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2137 (T367856)', diff saved to https://phabricator.wikimedia.org/P66472 and previous config saved to /var/cache/conftool/dbconfig/20240715-044723-marostegui.json [production]
04:47 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]
04:47 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]
04:13 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
04:13 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove IPV6 for dbproxy200[5-8] - pt1979@cumin2002" [production]
04:12 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove IPV6 for dbproxy200[5-8] - pt1979@cumin2002" [production]
04:09 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
02:11 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
02:11 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
02:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236 (T367856)', diff saved to https://phabricator.wikimedia.org/P66471 and previous config saved to /var/cache/conftool/dbconfig/20240715-021121-marostegui.json [production]
01:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P66470 and previous config saved to /var/cache/conftool/dbconfig/20240715-015613-marostegui.json [production]
01:41 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P66469 and previous config saved to /var/cache/conftool/dbconfig/20240715-014106-marostegui.json [production]
01:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236 (T367856)', diff saved to https://phabricator.wikimedia.org/P66467 and previous config saved to /var/cache/conftool/dbconfig/20240715-012559-marostegui.json [production]
2024-07-14 §
22:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1236 (T367856)', diff saved to https://phabricator.wikimedia.org/P66466 and previous config saved to /var/cache/conftool/dbconfig/20240714-223146-marostegui.json [production]
22:31 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1236.eqiad.wmnet with reason: Maintenance [production]
22:31 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1236.eqiad.wmnet with reason: Maintenance [production]