8401-8450 of 10000 results (99ms)
2022-12-02 ยง
18:05 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:03 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:01 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
18:00 <volans> performed git gc on all (auth)dns hosts in /srv/git/netbox_dns_snippets - T324334 [production]
17:36 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862944" [production]
16:56 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:53 <jnuche@deploy1002> Finished scap: testing k8s deployment (duration: 08m 35s) [production]
16:49 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:49 <bblack> (above agent runs completed on all text nodes for requestctl-for-misc patch) [production]
16:44 <jnuche@deploy1002> Started scap: testing k8s deployment [production]
16:44 <bblack> running agent on A:cp-text for https://gerrit.wikimedia.org/r/c/operations/puppet/+/863375 (requestctl for misc) [production]
16:29 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:28 <sukhe@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS buster [production]
16:21 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:03 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:02 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:59 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:55 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:48 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862998" [production]
15:47 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:43 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS buster [production]
15:40 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
15:39 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:36 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
15:33 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
15:30 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
15:29 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:28 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
15:22 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
15:22 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
15:13 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
15:12 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
15:06 <volans> run `git gc` on /srv/netbox-exports/dns.git on netbox[12]002 - T324334 [production]
14:48 <sukhe@cumin1001> START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS buster [production]
14:38 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns5004.wikimedia.org with OS buster [production]
12:09 <jynus> dropping all databases from db1133 [production]
11:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti5001.eqsin.wmnet [production]
11:16 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:16 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
11:12 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
11:02 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
10:57 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti5001.eqsin.wmnet [production]
10:56 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
10:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5001.eqsin.wmnet with reason: Remove from cluster for decom [production]
10:34 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti5001.eqsin.wmnet with reason: Remove from cluster for decom [production]
10:01 <vgutierrez> upload acme-chief 0.36 to apt.wm.o (bullseye) - T321309 [production]
09:58 <moritzm> installing publicsuffix updates from bullseye/buster point releases [production]
09:54 <moritzm> installing debootstrap updates from bullseye point release [production]
09:53 <moritzm> rebalance ganeti codfw/C T323222 [production]