751-800 of 10000 results (23ms)
2025-06-30 ยง
10:34 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:33 <jynus@cumin1003> DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for backup1003.eqiad.wmnet: Renew puppet certificate - jynus@cumin1003 [production]
10:33 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:30 <jmm@cumin1003> START - Cookbook sre.hosts.decommission for hosts ganeti2021.codfw.wmnet [production]
10:28 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:28 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:26 <mvernon@cumin1002> START - Cookbook sre.swift.check-dbs Checking container DBs of wikipedia-commons-local-thumb.6e [production]
10:25 <Emperor> repool codfw ms-swift after container DB repairs T383053 [production]
10:25 <mvernon@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw [production]
10:24 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be2076.codfw.wmnet [production]
10:24 <mvernon@cumin2002> START - Cookbook sre.hosts.remove-downtime for ms-be2076.codfw.wmnet [production]
10:22 <Emperor> repair wikipedia-commons-local-thumb.bb on ms-be2076 T383053 [production]
10:21 <mvernon@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be2076.codfw.wmnet with reason: container db repair [production]
10:17 <Emperor> repair wikipedia-commons-local-thumb.99 on ms-be2064 T383053 [production]
10:12 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
10:11 <jynus@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backup1003.eqiad.wmnet with reason: Maintenance and reboot [production]
10:06 <Emperor> repair wikipedia-commons-local-thumb.6e on ms-be2059 ms-be2058 ms-be2076 T383053 [production]
10:05 <mvernon@cumin2002> conftool action : set/pooled=false; selector: dnsdisc=swift,name=codfw [production]
10:05 <Emperor> depool codfw ms-swift for container DB repairs T383053 [production]
10:01 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet [production]
09:57 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet [production]
09:53 <moritzm> installing nginx security updates [production]
09:51 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [production]
09:51 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop: apply [production]
09:43 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
09:42 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop: apply [production]
09:42 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop: apply [production]
09:42 <dcaro> deleted the collected files-to-remove due to 100% capacity on nfs [paws]
09:39 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop: apply [production]
09:38 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/changeprop: apply [production]
09:23 <klausman@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-staging-ctrl2001.codfw.wmnet [production]
09:23 <klausman@cumin2002> START - Cookbook sre.hosts.remove-downtime for ml-staging-ctrl2001.codfw.wmnet [production]
09:23 <klausman@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet [production]
09:20 <klausman@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet [production]
09:20 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet [production]
09:19 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet [production]
09:18 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet [production]
09:14 <klausman@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ml-staging-ctrl2001.codfw.wmnet with reason: Shutting down for Ganeti migration [production]
09:04 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
09:04 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
08:50 <XioNoX> test routed ganeti compatible bird on ganeti2034/testvm2006 - T362392 [production]
08:47 <volans> uploaded debmonitor-server,python3-debmonitor_0.6.3 to apt.wikimedia.org bookworm-wikimedia [production]
08:39 <kostajh> UTC morning deploys done [production]
08:32 <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1163738|temp accounts: Enable temp account creation on further wikis (T397940)]] (duration: 09m 01s) [production]
08:26 <kharlan@deploy1003> kharlan, tchanders: Continuing with sync [production]
08:25 <kharlan@deploy1003> kharlan, tchanders: Backport for [[gerrit:1163738|temp accounts: Enable temp account creation on further wikis (T397940)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
08:23 <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1163738|temp accounts: Enable temp account creation on further wikis (T397940)]] [production]
08:19 <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1164506|initialiseSettings: set wgSecurePollUseMediaWikiNamespace = true for enwiki (T398080)]], [[gerrit:1164507|refactor unnecessary wmgSecurePollUseNamespace variable]] (duration: 16m 17s) [production]
08:12 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet [production]
08:11 <kharlan@deploy1003> novemlinguae, kharlan: Continuing with sync [production]