1-50 of 10000 results (20ms)
2025-04-04 §
00:51 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:41 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:35 <thcipriani> integration-agent-docker-1041 marked offline due to /srv disk space [releng]
00:34 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:23 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:23 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:14 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:11 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:10 <amastilovic@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
00:09 <Krinkle> Disable duplicate publishing noise from extension-MediaUploader, ref T143162, T389450 [releng]
2025-04-03 §
23:45 <tstarling@deploy1003> Finished scap sync-world: Backport for [[gerrit:1133592|Enable Codex and Multiblocks in German and Italian wiki (T377121)]] (duration: 15m 25s) [production]
23:38 <tstarling@deploy1003> hmonroy, tstarling: Continuing with sync [production]
23:35 <tstarling@deploy1003> hmonroy, tstarling: Backport for [[gerrit:1133592|Enable Codex and Multiblocks in German and Italian wiki (T377121)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
23:30 <tstarling@deploy1003> Started scap sync-world: Backport for [[gerrit:1133592|Enable Codex and Multiblocks in German and Italian wiki (T377121)]] [production]
22:43 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [admin]
22:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [admin]
22:26 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for all nodes [tools]
22:25 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 [tools]
22:25 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 [tools]
22:23 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 [tools]
22:22 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 [tools]
22:22 <root@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 [tools]
22:17 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 [tools]
22:16 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 [tools]
22:12 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 [tools]
22:12 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 [tools]
22:09 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 [tools]
22:07 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T381499) [admin]
22:02 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 [tools]
22:00 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T381499) [admin]
21:56 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T381499) [admin]
21:54 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [admin]
21:52 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 [tools]
21:49 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [admin]
21:49 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T381499) [admin]
21:46 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 [tools]
21:43 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T381499) [admin]
21:41 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 [tools]
21:38 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2056.codfw.wmnet with OS bullseye [production]
21:37 <James_F> Backport deploy done. [production]
21:36 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1134067|Revert "VE: Enable mobile insert menu everywhere except top 20 mobile VE wikipedias"]] (duration: 15m 28s) [production]
21:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T381499) [admin]
21:29 <jforrester@deploy1003> jforrester: Continuing with sync [production]
21:29 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cirrussearch* for ban cirrus nodes to prevent replication problems - bking@cumin2002 - T388610 [production]
21:29 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cirrussearch* for ban cirrus nodes to prevent replication problems - bking@cumin2002 - T388610 [production]
21:28 <jforrester@deploy1003> jforrester: Backport for [[gerrit:1134067|Revert "VE: Enable mobile insert menu everywhere except top 20 mobile VE wikipedias"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:21 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1134067|Revert "VE: Enable mobile insert menu everywhere except top 20 mobile VE wikipedias"]] [production]
21:19 <jforrester@deploy1003> Sync cancelled. [production]
21:13 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2056.codfw.wmnet with reason: host reimage [production]
21:09 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2056.codfw.wmnet with reason: host reimage [production]