401-450 of 10000 results (163ms)
2026-08-04 §
08:42 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2093.codfw.wmnet with reason: host reimage [production]
08:38 <dcausse@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
08:38 <dcausse@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
08:36 <mvernon@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2093.codfw.wmnet with reason: host reimage [production]
08:35 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
08:34 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
08:29 <dcausse@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
08:28 <dcausse@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
08:26 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1271.eqiad.wmnet [production]
08:23 <mvernon@cumin1003> START - Cookbook sre.hosts.reimage for host ms-be1094.eqiad.wmnet with OS trixie [production]
08:21 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.14 refs T430833 [production]
08:20 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1271.eqiad.wmnet [production]
08:20 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1270.eqiad.wmnet [production]
08:15 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1270.eqiad.wmnet [production]
08:15 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1269.eqiad.wmnet [production]
08:14 <mvernon@cumin2003> START - Cookbook sre.hosts.reimage for host ms-be2093.codfw.wmnet with OS trixie [production]
08:09 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1269.eqiad.wmnet [production]
08:09 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1268.eqiad.wmnet [production]
08:06 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2092.codfw.wmnet with OS trixie [production]
08:04 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1268.eqiad.wmnet [production]
08:04 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1267.eqiad.wmnet [production]
07:59 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1267.eqiad.wmnet [production]
07:58 <mvernon@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1093.eqiad.wmnet with OS trixie [production]
07:56 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1266.eqiad.wmnet [production]
07:55 <jynus> running extra backups to test db1285 T433826 [production]
07:51 <cwilliams@cumin1003> START - Cookbook sre.hosts.reboot-single for host db1266.eqiad.wmnet [production]
07:43 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2092.codfw.wmnet with reason: host reimage [production]
07:39 <mvernon@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage [production]
07:33 <mvernon@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2092.codfw.wmnet with reason: host reimage [production]
07:32 <mvernon@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1093.eqiad.wmnet with reason: host reimage [production]
07:29 <jynus> running extra backups to test db1265 T433825 [production]
07:12 <mvernon@cumin2003> START - Cookbook sre.hosts.reimage for host ms-be2092.codfw.wmnet with OS trixie [production]
07:11 <mvernon@cumin1003> START - Cookbook sre.hosts.reimage for host ms-be1093.eqiad.wmnet with OS trixie [production]
06:50 <slyngshede@dns1004> END - running authdns-update [production]
06:48 <slyngshede@dns1004> START - running authdns-update [production]
04:02 <mwpresync@deploy1003> Pruned MediaWiki: 1.47.0-wmf.11 (duration: 02m 29s) [production]
03:38 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.47.0-wmf.14 refs T430833 (duration: 32m 57s) [production]
03:23 <cjming@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [production]
03:22 <cjming@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [production]
03:05 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.47.0-wmf.14 refs T430833 [production]
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 32s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:45 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1320296|Fix InstrumentConstructiveEdits script (T431493)]] (duration: 06m 20s) [production]
00:41 <cjming@deploy1003> cjming: Continuing with deployment [production]
00:41 <cjming@deploy1003> cjming: Backport for [[gerrit:1320296|Fix InstrumentConstructiveEdits script (T431493)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
00:39 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1320296|Fix InstrumentConstructiveEdits script (T431493)]] [production]
2026-08-03 §
23:58 <rzl@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
23:57 <rzl@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
23:29 <robh@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp5021.eqsin.wmnet [production]
23:29 <robh@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp5021.eqsin.wmnet [production]