1-50 of 10000 results (23ms)
2025-07-05 §
00:47 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 [tools]
00:31 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 [tools]
00:31 <andrewbogott> restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs [tools]
2025-07-04 §
21:45 <Krinkle> T289318: Change stream_config_uri in Hiera (Horizon instance config for deployment-eventgate-4 and deployment-eventstreams-2 ) from https://meta.wikimedia.beta.wmflabs.org/w/api.php?action=streamconfigs to https://meta.wikimedia.beta.wmcloud.org/w/api.php?action=streamconfigs [releng]
21:45 <Krinkle> T289318: Change profile::cache::varnish::frontend::fe_vcl_config/static_host in Hiera (Horizon puppet prefix for cache-text and cache-upload) from en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org [releng]
21:41 <Krinkle> Change profile::docker::runner::service_defs/mediawiki-services-cxserver/mwapi_req/host in Horizon (Hiera puppet prefix) from en.wikipedia.beta.wmflabs.org to en.wikipedia.beta.wmcloud.org. T289318 [releng]
21:39 <krinkle@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166438|beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] (duration: 18m 12s) [production]
21:39 <Krinkle> Change profile::docker::runner::service_defs/mediawiki-services-push-notifications/mwapi_req/host in Horizon (Hiera puppet prefix) from meta.wikimedia.beta.wmflabs.org to meta.wikimedia.beta.wmcloud.org. T289318 [releng]
21:33 <krinkle@deploy1003> krinkle: Continuing with sync [production]
21:23 <krinkle@deploy1003> krinkle: Backport for [[gerrit:1166438|beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:21 <krinkle@deploy1003> Started scap sync-world: Backport for [[gerrit:1166438|beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org (T289318)]] [production]
20:32 <krinkle@deploy1003> Finished scap sync-world: Backport for [[gerrit:1165989|beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999|beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] (duration: 94m 52s) [production]
20:26 <krinkle@deploy1003> krinkle: Continuing with sync [production]
18:59 <krinkle@deploy1003> krinkle: Backport for [[gerrit:1165989|beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999|beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:57 <krinkle@deploy1003> Started scap sync-world: Backport for [[gerrit:1165989|beta: Include allowance for wmcloud.org in wgGraphAllowedDomains (T289318)]], [[gerrit:1165999|beta: Change Beta wikidata canonical to beta.wmcloud.org (T289318)]] [production]
15:14 <vgutierrez> fetch haproxy 2.8.15 on thirdparty/haproxy28 component for bullseye-wikimedia (apt.wm.o) [production]
14:56 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 [tools]
14:46 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2043.codfw.wmnet with OS bullseye [production]
14:44 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 [tools]
14:40 <stevemunene@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye [production]
14:36 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host cp2043.codfw.wmnet with OS bullseye [production]
14:29 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
14:20 <vgutierrez> repooling cp7006 [production]
14:20 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
14:12 <vgutierrez> depooling cp7006 for testing purposes [production]
14:09 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
14:06 <stevemunene@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye [production]
14:01 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
13:49 <hashar> gerrit: deleted project glam/gwtoolset | Created October 11st 2012 and has never been used [releng]
13:30 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 [tools]
13:24 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 [tools]
13:24 <hashar> gerrit: changed `All-Projects` default submit strategy to `Rebase if Necessary`. Does not affect mediawiki/* or operations/* among others # T390719 [releng]
13:15 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
13:08 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
12:59 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
12:51 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
12:51 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project wikidata-deleted in eqiad1 (T398254) [wikidata-deleted]
12:49 <taavi@cloudcumin1001> START - Cookbook wmcs.vps.create_project for project wikidata-deleted in eqiad1 (T398254) [wikidata-deleted]
12:31 <vgutierrez> repool cp7006 [production]
12:31 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7006.magru.wmnet [production]
12:31 <vgutierrez@cumin1002> START - Cookbook sre.hosts.remove-downtime for cp7006.magru.wmnet [production]
12:11 <mfossati@deploy1003> Finished deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 (duration: 00m 49s) [production]
12:11 <mfossati@deploy1003> Started deploy [airflow-dags/platform_eng@38ba3ec]: bump section topics to v1.8.0 [production]
11:08 <cmooney@cumin1003> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003 [production]
11:05 <cmooney@cumin1003> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin[1002-1003].eqiad.wmnet with reason: Release v10.0.2 with ibgp function in plugin - cmooney@cumin1003 [production]
10:56 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 32 hosts with reason: maintenance [production]
10:51 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2006.codfw.wmnet with OS bookworm [production]
10:43 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2203,2212].codfw.wmnet with reason: Maintenance [production]
10:41 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm [production]
10:27 <cgoubert@deploy1003> Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 07s) [production]