1-50 of 10000 results (78ms)
2025-09-18 ยง
14:34 <moritzm> upgrading Envoy on cloudweb hosts T403663 [production]
14:33 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1129894|Graph: Use new placeholder i18n from WikimediaMessages (T362317)]] (duration: 11m 40s) [production]
14:28 <jforrester@deploy1003> jforrester: Continuing with sync [production]
14:26 <elukey@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2048.codfw.wmnet'] [production]
14:26 <elukey@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2048.codfw.wmnet'] [production]
14:26 <andrew@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcephosd1022.eqiad.wmnet with OS bookworm [production]
14:24 <jforrester@deploy1003> jforrester: Backport for [[gerrit:1129894|Graph: Use new placeholder i18n from WikimediaMessages (T362317)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
14:22 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1129894|Graph: Use new placeholder i18n from WikimediaMessages (T362317)]] [production]
14:18 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1189491|test2wiki: Enable Wikifunctions client mode here too (T397401)]] (duration: 11m 16s) [production]
14:17 <elukey@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp2048.codfw.wmnet'] [production]
14:16 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=93) for host cp2049.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:16 <elukey@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2048.codfw.wmnet'] [production]
14:12 <jforrester@deploy1003> jforrester: Continuing with sync [production]
14:11 <jforrester@deploy1003> jforrester: Backport for [[gerrit:1189491|test2wiki: Enable Wikifunctions client mode here too (T397401)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
14:06 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1189491|test2wiki: Enable Wikifunctions client mode here too (T397401)]] [production]
14:05 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1189492|Actiave thwikimedia (T400001)]], [[gerrit:1189494|Update interwiki cache]] (duration: 08m 39s) [production]
13:59 <zabe@deploy1003> zabe: Continuing with sync [production]
13:58 <zabe@deploy1003> zabe: Backport for [[gerrit:1189492|Actiave thwikimedia (T400001)]], [[gerrit:1189494|Update interwiki cache]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:56 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1189492|Actiave thwikimedia (T400001)]], [[gerrit:1189494|Update interwiki cache]] [production]
13:52 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1188894|Initial configuration for thwikimedia (T400001)]] (duration: 10m 34s) [production]
13:51 <moritzm> imported imposm 0.14.1-3 (cherrypick of upstream fix to hopefully fix deadlock in OSM import) T381565 [production]
13:51 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:50 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
13:49 <btullis@cumin1003> START - Cookbook sre.dns.netbox [production]
13:47 <zabe@deploy1003> zabe: Continuing with sync [production]
13:46 <zabe@deploy1003> zabe: Backport for [[gerrit:1188894|Initial configuration for thwikimedia (T400001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:42 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1188894|Initial configuration for thwikimedia (T400001)]] [production]
13:38 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1189486|Activate mswikiquote (T404698)]] (duration: 10m 58s) [production]
13:37 <ladsgroup@cumin1003> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for all core sections [production]
13:36 <btullis@cumin1003> START - Cookbook sre.hosts.decommission for hosts an-backup-namenode[1001-1002].eqiad.wmnet [production]
13:33 <zabe@deploy1003> zabe: Continuing with sync [production]
13:32 <zabe@deploy1003> zabe: Backport for [[gerrit:1189486|Activate mswikiquote (T404698)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:27 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1189486|Activate mswikiquote (T404698)]] [production]
13:25 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1188892|Initial configuration for mswikiquote (T404698)]] (duration: 07m 56s) [production]
13:24 <kart_> Updated Recommendation API to 2025-09-15-194552-production (T404223. T404448. T400562) [production]
13:21 <kartik@deploy1003> helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
13:20 <zabe@deploy1003> zabe: Continuing with sync [production]
13:19 <zabe@deploy1003> zabe: Backport for [[gerrit:1188892|Initial configuration for mswikiquote (T404698)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:17 <ladsgroup@cumin1003> START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for all core sections [production]
13:17 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1188892|Initial configuration for mswikiquote (T404698)]] [production]
13:16 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1189267|Release CampaignEvents extension to Wikimedia Commons - Sept 18 (T403667)]] (duration: 11m 15s) [production]
13:16 <kartik@deploy1003> helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
13:11 <zabe@deploy1003> zabe, cmelo: Continuing with sync [production]
13:11 <cmooney@cumin1003> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 62856 [production]
13:10 <zabe@deploy1003> zabe, cmelo: Backport for [[gerrit:1189267|Release CampaignEvents extension to Wikimedia Commons - Sept 18 (T403667)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:05 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1189267|Release CampaignEvents extension to Wikimedia Commons - Sept 18 (T403667)]] [production]
13:04 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
13:04 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
13:01 <cmooney@cumin1003> START - Cookbook sre.network.peering with action 'configure' for AS: 62856 [production]
12:49 <kartik@deploy1003> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]