51-100 of 10000 results (79ms)
2025-09-18 ยง
16:20 <jasmine@cumin1003> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad [production]
16:19 <jasmine@cumin1003> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad [production]
16:19 <jasmine@cumin1003> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad [production]
16:19 <jasmine@cumin1003> [DRY-RUN] MediaWiki read-only period starts at: 2025-09-18 16:19:18.465479 [production]
16:19 <jasmine@cumin1003> START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad [production]
16:17 <jasmine@cumin1003> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad [production]
16:17 <jasmine@cumin1003> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad [production]
16:17 <tchin@deploy1003> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
16:16 <jasmine@cumin1003> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from codfw to eqiad [production]
16:16 <tchin@deploy1003> helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [production]
16:16 <jasmine@deploy1003> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T399891 [production]
16:13 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:13 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:11 <jasmine@cumin1003> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad [production]
16:10 <jasmine@cumin1003> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from codfw to eqiad [production]
16:10 <jasmine@cumin1003> START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad [production]
16:03 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:01 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:41 <kharlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1189500|hCaptcha: Log hcaptcha.execute() events (T402767)]] (duration: 12m 20s) [production]
15:39 <jnuche@deploy1003> Finished deploy [releng/jenkins-deploy@b41bbe7] (releasing): Update Jenkins version (duration: 00m 42s) [production]
15:39 <jnuche@deploy1003> Started deploy [releng/jenkins-deploy@b41bbe7] (releasing): Update Jenkins version [production]
15:36 <kharlan@deploy1003> kharlan: Continuing with sync [production]
15:35 <kharlan@deploy1003> kharlan: Backport for [[gerrit:1189500|hCaptcha: Log hcaptcha.execute() events (T402767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
15:29 <kharlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1189500|hCaptcha: Log hcaptcha.execute() events (T402767)]] [production]
15:24 <cmooney@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on 7 hosts with reason: reboot cr1-codfw as requested by Juniper [production]
15:24 <zabe> zabe@deploy1003:~$ mwscript createAndPromote.php --wiki=thwikimedia --bureaucrat --sysop --reason="T400001" Sarawut.Kha REDACTED [production]
15:23 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:23 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:22 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps2011.codfw.wmnet with OS bookworm [production]
15:21 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:21 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:18 <tchin@deploy1003> helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
15:17 <tchin@deploy1003> helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [production]
15:07 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1022.eqiad.wmnet with OS bookworm [production]
15:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2011.codfw.wmnet with reason: host reimage [production]
14:58 <topranks> drain cr1-codfw of traffic before work to test power cupplies T401937 [production]
14:56 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2011.codfw.wmnet with reason: host reimage [production]
14:50 <jnuche@deploy1003> Finished deploy [releng/jenkins-deploy@b41bbe7] (releasing): Test deploy (duration: 00m 30s) [production]
14:49 <jnuche@deploy1003> Started deploy [releng/jenkins-deploy@b41bbe7] (releasing): Test deploy [production]
14:49 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1022.eqiad.wmnet with reason: host reimage [production]
14:46 <jnuche@deploy1003> Finished deploy [releng/jenkins-deploy@b41bbe7] (releasing): Test deploy (duration: 00m 19s) [production]
14:46 <jnuche@deploy1003> Started deploy [releng/jenkins-deploy@b41bbe7] (releasing): Test deploy [production]
14:43 <andrew@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1022.eqiad.wmnet with reason: host reimage [production]
14:38 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-backup-namenode[1001-1002].eqiad.wmnet [production]
14:38 <btullis@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:38 <btullis@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-namenode[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [production]
14:37 <btullis@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-namenode[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [production]
14:37 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host maps2011.codfw.wmnet with OS bookworm [production]
14:34 <moritzm> upgrading Envoy on cloudweb hosts T403663 [production]
14:33 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1129894|Graph: Use new placeholder i18n from WikimediaMessages (T362317)]] (duration: 11m 40s) [production]