901-950 of 10000 results (127ms)
2024-08-06 §
05:41 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20240729/ using stat1009.eqiad.wmnet) [production]
04:39 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-reload reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20240729/ using stat1009.eqiad.wmnet) [production]
04:38 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-reload reloading wikidata_main on wdqs1021.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20240729/ using stat1009.eqiad.wmnet) [production]
04:37 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host (duration: 00m 09s) [production]
04:37 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host [production]
04:36 <ryankemper@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1023.eqiad.wmnet with OS bullseye [production]
04:36 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host (duration: 00m 09s) [production]
04:36 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host [production]
04:01 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.14 (duration: 00m 58s) [production]
03:47 <mwpresync@deploy1003> Finished scap: testwikis to 1.43.0-wmf.17 refs T366962 (duration: 45m 05s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.17 refs T366962 [production]
2024-08-05 §
20:47 <cjming> end of UTC late backport window [production]
20:44 <cjming@deploy1003> Finished scap: Backport for [[gerrit:1059374|Add wikibase client interaction stream to Event Logging (T370045)]] (duration: 22m 52s) [production]
20:39 <cjming@deploy1003> cjming, joelyrookewmde: Continuing with sync [production]
20:23 <cjming@deploy1003> cjming, joelyrookewmde: Backport for [[gerrit:1059374|Add wikibase client interaction stream to Event Logging (T370045)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:21 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1059374|Add wikibase client interaction stream to Event Logging (T370045)]] [production]
19:29 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_main on wdqs1021.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20240729/ using stat1009.eqiad.wmnet) [production]
19:14 <otto@deploy1003> Finished scap: Backport for [[gerrit:1059900|eventbus: enable instrumentation on all wikis (T363587)]] (duration: 07m 08s) [production]
19:10 <otto@deploy1003> otto: Continuing with sync [production]
19:09 <otto@deploy1003> otto: Backport for [[gerrit:1059900|eventbus: enable instrumentation on all wikis (T363587)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
19:07 <otto@deploy1003> Started scap sync-world: Backport for [[gerrit:1059900|eventbus: enable instrumentation on all wikis (T363587)]] [production]
18:56 <dancy@deploy1003> sync-world aborted: testing scap 4.96.0 (duration: 03m 11s) [production]
18:53 <dancy@deploy1003> Started scap sync-world: testing scap 4.96.0 [production]
18:52 <dancy@deploy1003> Installation of scap version "4.96.0" completed for 211 hosts [production]
18:52 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-reload reloading wikidata_main on wdqs1021.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20240729/ using stat1009.eqiad.wmnet) [production]
18:52 <dancy@deploy1003> Installing scap version "4.96.0" for 211 hosts [production]
18:27 <dancy@deploy1003> Started scap sync-world: testing updates to repos/releng/release/make-container-image [production]
17:28 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1023.eqiad.wmnet with reason: host reimage [production]
17:25 <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1023.eqiad.wmnet with reason: host reimage [production]
17:04 <ryankemper@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1023.eqiad.wmnet with OS bullseye [production]
16:52 <dcausse@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:52 <dcausse@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:52 <mutante> DNS - added new project language 'bdr' - West Coast Bajau - https://en.wikipedia.org/wiki/Sama%E2%80%93Bajaw_languages - T371757 [production]
16:36 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2008.codfw.wmnet with OS bookworm [production]
16:33 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2007.codfw.wmnet with OS bookworm [production]
16:20 <dcausse@deploy1003> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:19 <dcausse@deploy1003> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:18 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on prometheus2008.codfw.wmnet with reason: host reimage [production]
16:15 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on prometheus2007.codfw.wmnet with reason: host reimage [production]
16:12 <filippo@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on prometheus2008.codfw.wmnet with reason: host reimage [production]
16:11 <filippo@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on prometheus2007.codfw.wmnet with reason: host reimage [production]
16:11 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:10 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:53 <filippo@cumin1002> START - Cookbook sre.hosts.reimage for host prometheus2008.codfw.wmnet with OS bookworm [production]
15:42 <filippo@cumin1002> START - Cookbook sre.hosts.reimage for host prometheus2007.codfw.wmnet with OS bookworm [production]
15:41 <filippo@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host prometheus2007.codfw.wmnet with OS bookworm [production]
15:40 <filippo@cumin1002> START - Cookbook sre.hosts.reimage for host prometheus2007.codfw.wmnet with OS bookworm [production]
15:39 <filippo@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host prometheus2007.codfw.wmnet with OS bookworm [production]
15:39 <elukey@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
15:38 <elukey@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]