1301-1350 of 10000 results (20ms)
2025-06-30 ยง
12:10 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[1078,1080].eqiad.wmnet [production]
12:10 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be[1078,1080].eqiad.wmnet [production]
12:07 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
12:05 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[1078,1080].eqiad.wmnet with reason: container db repair [production]
12:05 <Emperor> repair wikipedia-commons-local-thumb.ea on ms-be10[78,80] T383053 [production]
12:04 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[1078-1079,1085].eqiad.wmnet [production]
12:04 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be[1078-1079,1085].eqiad.wmnet [production]
12:01 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [production]
12:00 <btullis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [production]
11:59 <Emperor> repair wikipedia-commons-local-thumb.d3 on ms-be10[78,79,85] T383053 [production]
11:59 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[1078-1079,1085].eqiad.wmnet with reason: container db repair [production]
11:58 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti5004.eqsin.wmnet with OS bookworm [production]
11:57 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[1066,1087,1090].eqiad.wmnet [production]
11:57 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be[1066,1087,1090].eqiad.wmnet [production]
11:56 <hnowlan> restarting pybal on A:lvs-low-traffic-eqiad [production]
11:56 <jmm@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti5004.eqsin.wmnet with reason: reimage [production]
11:52 <moritzm> installing mongo-c-driver security updates [production]
11:52 <Emperor> repair wikipedia-commons-local-thumb.b7 ms-be10[66,87,90] T383053 [production]
11:52 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[1066,1087,1090].eqiad.wmnet with reason: container db repair [production]
11:50 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [production]
11:50 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[1067,1070,1089].eqiad.wmnet [production]
11:50 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be[1067,1070,1089].eqiad.wmnet [production]
11:50 <hnowlan> restarting pybal on lvs-secondary-eqiad [production]
11:47 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
11:45 <Emperor> repair wikipedia-commons-local-thumb.79 on ms-be10[70,67,89] T383053 [production]
11:45 <dcaro> added a new global alert when nfs space is >90% [metricsinfra]
11:45 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[1067,1070,1089].eqiad.wmnet with reason: container db repair [production]
11:44 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[1063,1074,1083].eqiad.wmnet [production]
11:43 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be[1063,1074,1083].eqiad.wmnet [production]
11:41 <phuedx@deploy1003> Finished scap sync-world: Backport for [[gerrit:1164984|Revert "Clean up EventBus and jobs config"]] (duration: 17m 19s) [production]
11:39 <Emperor> repair wikipedia-commons-local-thumb.6b on ms-be10[63,74,83] T383053 [production]
11:38 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be[1063,1074,1083].eqiad.wmnet with reason: container db repair [production]
11:36 <phuedx@deploy1003> gmodena, phuedx: Continuing with sync [production]
11:29 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet [production]
11:29 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet [production]
11:26 <phuedx@deploy1003> gmodena, phuedx: Backport for [[gerrit:1164984|Revert "Clean up EventBus and jobs config"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
11:24 <phuedx@deploy1003> Started scap sync-world: Backport for [[gerrit:1164984|Revert "Clean up EventBus and jobs config"]] [production]
11:22 <mvernon@cumin2002> conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad [production]
11:21 <Emperor> depool eqiad ms-swift for container DB repairs T383053 [production]
11:20 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet [production]
11:13 <taavi@dns1004> END - running authdns-update [production]
11:12 <urbanecm> Start GrowthExperiments:fixLinkRecommendationData --wiki=enwiki --db-table --force (T386867) [production]
11:11 <taavi@dns1004> START - running authdns-update [production]
11:10 <taavi@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:10 <taavi@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add cloudvps ns-recursor v6 addresses - taavi@cumin1003" [production]
11:09 <taavi@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add cloudvps ns-recursor v6 addresses - taavi@cumin1003" [production]
11:07 <jmm@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2022.codfw.wmnet [production]
11:07 <jmm@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:05 <vgutierrez> repool cp7001 [production]
11:05 <jmm@cumin1003> START - Cookbook sre.dns.netbox [production]