6501-6550 of 10000 results (81ms)
2022-12-02 ยง
19:36 <volans> fixed git checkout permissions T324334 [production]
19:11 <sukhe> restart pybal on lvs5004 [production]
19:07 <mutante> gitlab-runner* - upgrading gitlab-runner package version [production]
18:55 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 863383" [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs5001.eqsin.wmnet [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:51 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:49 <sukhe@cumin2002> START - Cookbook sre.dns.netbox [production]
18:44 <sukhe@cumin2002> START - Cookbook sre.hosts.decommission for hosts lvs5001.eqsin.wmnet [production]
18:22 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs5001.eqsin.wmnet with reason: downtimed, in the process of decom [production]
18:21 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on lvs5001.eqsin.wmnet with reason: downtimed, in the process of decom [production]
18:20 <sukhe> decomm lvs5001: restarting pybal [production]
18:14 <sukhe> cr[23]-eqsin*: set routing-options static route 103.102.166.224/28 next-hop 10.132.0.39 [production]
18:05 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:05 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:03 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:01 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
18:00 <volans> performed git gc on all (auth)dns hosts in /srv/git/netbox_dns_snippets - T324334 [production]
17:36 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862944" [production]
16:56 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:53 <jnuche@deploy1002> Finished scap: testing k8s deployment (duration: 08m 35s) [production]
16:49 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:49 <bblack> (above agent runs completed on all text nodes for requestctl-for-misc patch) [production]
16:44 <jnuche@deploy1002> Started scap: testing k8s deployment [production]
16:44 <bblack> running agent on A:cp-text for https://gerrit.wikimedia.org/r/c/operations/puppet/+/863375 (requestctl for misc) [production]
16:29 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:28 <sukhe@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS buster [production]
16:21 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:03 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:02 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:59 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:55 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:48 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862998" [production]
15:47 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:43 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS buster [production]
15:40 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
15:39 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:36 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
15:33 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
15:30 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
15:29 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:28 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
15:22 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
15:22 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
15:13 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
15:12 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns5004.wikimedia.org with reason: host reimage [production]
15:06 <volans> run `git gc` on /srv/netbox-exports/dns.git on netbox[12]002 - T324334 [production]
14:48 <sukhe@cumin1001> START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS buster [production]