3401-3450 of 10000 results (82ms)
2022-12-05 §
06:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db2173 (re)pooling @ 5%: After HW issues', diff saved to https://phabricator.wikimedia.org/P42221 and previous config saved to /var/cache/conftool/dbconfig/20221205-063130-root.json [production]
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 to dbctl (depooled)', diff saved to https://phabricator.wikimedia.org/P42220 and previous config saved to /var/cache/conftool/dbconfig/20221205-063020-marostegui.json [production]
06:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P42219 and previous config saved to /var/cache/conftool/dbconfig/20221205-061735-root.json [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db2173 (re)pooling @ 1%: After HW issues', diff saved to https://phabricator.wikimedia.org/P42218 and previous config saved to /var/cache/conftool/dbconfig/20221205-061625-root.json [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1132', diff saved to https://phabricator.wikimedia.org/P42217 and previous config saved to /var/cache/conftool/dbconfig/20221205-061616-marostegui.json [production]
2022-12-04 §
04:19 <TheresNoTime> T302486 : `[samtar@mwmaint1002 ~]$ mwscript maintenance/fixMergeHistoryCorruption.php --wiki enwiki --dry-run --ns 828` [production]
2022-12-03 §
00:17 <cwhite> draining shards from logstash1010, logstash1033, logstash1034, logstash1035 - T321410 [production]
2022-12-02 §
19:42 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:42 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force run after a permission problem - volans@cumin1001" [production]
19:41 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force run after a permission problem - volans@cumin1001" [production]
19:39 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
19:38 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:37 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
19:36 <volans> fixed git checkout permissions T324334 [production]
19:11 <sukhe> restart pybal on lvs5004 [production]
19:07 <mutante> gitlab-runner* - upgrading gitlab-runner package version [production]
18:55 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 863383" [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs5001.eqsin.wmnet [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:53 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:51 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs5001.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:49 <sukhe@cumin2002> START - Cookbook sre.dns.netbox [production]
18:44 <sukhe@cumin2002> START - Cookbook sre.hosts.decommission for hosts lvs5001.eqsin.wmnet [production]
18:22 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs5001.eqsin.wmnet with reason: downtimed, in the process of decom [production]
18:21 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on lvs5001.eqsin.wmnet with reason: downtimed, in the process of decom [production]
18:20 <sukhe> decomm lvs5001: restarting pybal [production]
18:14 <sukhe> cr[23]-eqsin*: set routing-options static route 103.102.166.224/28 next-hop 10.132.0.39 [production]
18:05 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:05 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:03 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Test run after git gc - volans@cumin1001" [production]
18:01 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
18:00 <volans> performed git gc on all (auth)dns hosts in /srv/git/netbox_dns_snippets - T324334 [production]
17:36 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862944" [production]
16:56 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:53 <jnuche@deploy1002> Finished scap: testing k8s deployment (duration: 08m 35s) [production]
16:49 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:49 <bblack> (above agent runs completed on all text nodes for requestctl-for-misc patch) [production]
16:44 <jnuche@deploy1002> Started scap: testing k8s deployment [production]
16:44 <bblack> running agent on A:cp-text for https://gerrit.wikimedia.org/r/c/operations/puppet/+/863375 (requestctl for misc) [production]
16:29 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:28 <sukhe@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS buster [production]
16:21 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
16:03 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
16:02 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:59 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [production]
15:55 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
15:48 <sukhe> homer "cr*-eqsin*" commit "running homer for Gerrit: 862998" [production]
15:47 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
15:43 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5004.wikimedia.org with OS buster [production]
15:40 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]