551-600 of 10000 results (168ms)
2025-05-28 ยง
12:32 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T395241)', diff saved to https://phabricator.wikimedia.org/P76593 and previous config saved to /var/cache/conftool/dbconfig/20250528-123230-fceratto.json [production]
12:31 <btullis@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on an-druid1003.eqiad.wmnet with reason: Cold booting to address disk failure [production]
12:31 <elukey@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
12:30 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P76592 and previous config saved to /var/cache/conftool/dbconfig/20250528-123012-fceratto.json [production]
12:19 <jynus@cumin1002> DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 4:00:00 on backup[2010-2011].codfw.wmnet with reason: Downtime hosts for reboot [production]
12:17 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P76591 and previous config saved to /var/cache/conftool/dbconfig/20250528-121723-fceratto.json [production]
12:15 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb1001.eqiad.wmnet with reason: host reimage [production]
12:15 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P76590 and previous config saved to /var/cache/conftool/dbconfig/20250528-121505-fceratto.json [production]
12:12 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb1001.eqiad.wmnet with reason: host reimage [production]
12:11 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=93) for new host ncredir7003.magru.wmnet [production]
12:11 <jmm@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
12:05 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
12:05 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ncredir7003.magru.wmnet on all recursors [production]
12:05 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache ncredir7003.magru.wmnet on all recursors [production]
12:05 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:05 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ncredir7003.magru.wmnet - jmm@cumin2002" [production]
12:04 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ncredir7003.magru.wmnet - jmm@cumin2002" [production]
12:02 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P76589 and previous config saved to /var/cache/conftool/dbconfig/20250528-120215-fceratto.json [production]
12:01 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
12:01 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host ncredir7003.magru.wmnet [production]
11:59 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T395241)', diff saved to https://phabricator.wikimedia.org/P76588 and previous config saved to /var/cache/conftool/dbconfig/20250528-115958-fceratto.json [production]
11:57 <jynus@cumin1002> DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 4:00:00 on backup[1010-1011].eqiad.wmnet with reason: Downtime hosts for reboot [production]
11:52 <stevemunene@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1163-1165].eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB [production]
11:50 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2177 (T395241)', diff saved to https://phabricator.wikimedia.org/P76587 and previous config saved to /var/cache/conftool/dbconfig/20250528-115012-fceratto.json [production]
11:50 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
11:49 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T395241)', diff saved to https://phabricator.wikimedia.org/P76586 and previous config saved to /var/cache/conftool/dbconfig/20250528-114944-fceratto.json [production]
11:47 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T395241)', diff saved to https://phabricator.wikimedia.org/P76585 and previous config saved to /var/cache/conftool/dbconfig/20250528-114708-fceratto.json [production]
11:43 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 141626 [production]
11:43 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'email' for AS: 141626 [production]
11:41 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1180 (T395241)', diff saved to https://phabricator.wikimedia.org/P76584 and previous config saved to /var/cache/conftool/dbconfig/20250528-114149-fceratto.json [production]
11:41 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
11:41 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T395241)', diff saved to https://phabricator.wikimedia.org/P76583 and previous config saved to /var/cache/conftool/dbconfig/20250528-114124-fceratto.json [production]
11:38 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 137409 [production]
11:34 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P76582 and previous config saved to /var/cache/conftool/dbconfig/20250528-113437-fceratto.json [production]
11:26 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P76581 and previous config saved to /var/cache/conftool/dbconfig/20250528-112617-fceratto.json [production]
11:25 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'configure' for AS: 137409 [production]
11:20 <taavi@cumin1002> START - Cookbook sre.hosts.reimage for host cloudlb1001.eqiad.wmnet with OS bookworm [production]
11:19 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P76580 and previous config saved to /var/cache/conftool/dbconfig/20250528-111931-fceratto.json [production]
11:11 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P76579 and previous config saved to /var/cache/conftool/dbconfig/20250528-111109-fceratto.json [production]
11:07 <marostegui@cumin1002> dbctl commit (dc=all): 'es2035 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76578 and previous config saved to /var/cache/conftool/dbconfig/20250528-110750-root.json [production]
11:04 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T395241)', diff saved to https://phabricator.wikimedia.org/P76577 and previous config saved to /var/cache/conftool/dbconfig/20250528-110423-fceratto.json [production]
10:56 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T395241)', diff saved to https://phabricator.wikimedia.org/P76576 and previous config saved to /var/cache/conftool/dbconfig/20250528-105602-fceratto.json [production]
10:53 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2156 (T395241)', diff saved to https://phabricator.wikimedia.org/P76575 and previous config saved to /var/cache/conftool/dbconfig/20250528-105346-fceratto.json [production]
10:53 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
10:53 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T395241)', diff saved to https://phabricator.wikimedia.org/P76574 and previous config saved to /var/cache/conftool/dbconfig/20250528-105320-fceratto.json [production]
10:52 <marostegui@cumin1002> dbctl commit (dc=all): 'es2035 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76573 and previous config saved to /var/cache/conftool/dbconfig/20250528-105245-root.json [production]
10:49 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1168 (T395241)', diff saved to https://phabricator.wikimedia.org/P76572 and previous config saved to /var/cache/conftool/dbconfig/20250528-104928-fceratto.json [production]
10:49 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
10:49 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T395241)', diff saved to https://phabricator.wikimedia.org/P76571 and previous config saved to /var/cache/conftool/dbconfig/20250528-104901-fceratto.json [production]
10:40 <ayounsi@dns1004> END - running authdns-update [production]