|
2026-06-23
ยง
|
| 05:35 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2038.codfw.wmnet with reason: host reimage |
[production] |
| 05:31 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on es2038.codfw.wmnet with reason: host reimage |
[production] |
| 05:13 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.reimage for host es2038.codfw.wmnet with OS trixie |
[production] |
| 05:12 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2038: Upgrading es2038.codfw.wmnet |
[production] |
| 05:12 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es2038: Upgrading es2038.codfw.wmnet |
[production] |
| 05:12 |
<marostegui@cumin1003> |
dbmaint on es7@codfw T429463 |
[production] |
| 05:12 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.major-upgrade |
[production] |
| 05:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool es2038 T429794', diff saved to https://phabricator.wikimedia.org/P94332 and previous config saved to /var/cache/conftool/dbconfig/20260623-051012-marostegui.json |
[production] |
| 05:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote es2039 to es7 primary T429794', diff saved to https://phabricator.wikimedia.org/P94331 and previous config saved to /var/cache/conftool/dbconfig/20260623-050758-marostegui.json |
[production] |
| 05:07 |
<marostegui> |
Starting es7 codfw failover from es2038 to es2039 - T429794 |
[production] |
| 05:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set es2039 with weight 0 T429794', diff saved to https://phabricator.wikimedia.org/P94330 and previous config saved to /var/cache/conftool/dbconfig/20260623-050137-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Primary switchover es7 T429794 |
[production] |
| 04:02 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.47.0-wmf.5 (duration: 02m 37s) |
[production] |
| 03:44 |
<mwpresync@deploy1003> |
Finished scap sync-world: testwikis to 1.47.0-wmf.8 refs T423917 (duration: 39m 23s) |
[production] |
| 03:08 |
<mwpresync@deploy1003> |
Started scap sync-world: testwikis to 1.47.0-wmf.8 refs T423917 |
[production] |
| 02:08 |
<mwpresync@deploy1003> |
Finished scap build-images: Publishing wmf/next image (duration: 07m 25s) |
[production] |
| 02:00 |
<mwpresync@deploy1003> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 01:41 |
<egardner@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1304859|Remove multimediaviewer-beta]] (duration: 11m 14s) |
[production] |
| 01:34 |
<egardner@deploy1003> |
egardner, ksarabia: Continuing with deployment |
[production] |
| 01:34 |
<egardner@deploy1003> |
egardner, ksarabia: Backport for [[gerrit:1304859|Remove multimediaviewer-beta]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 01:30 |
<egardner@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1304859|Remove multimediaviewer-beta]] |
[production] |
| 01:24 |
<egardner@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1304914|Inject service RepoGroup into Hooks]], [[gerrit:1304909|MMV Beta Viewer: Improve loading/navigation UX (T429193)]], [[gerrit:1304910|Take the feature out of beta (T429509)]] (duration: 33m 22s) |
[production] |
| 01:12 |
<egardner@deploy1003> |
egardner: Continuing with deployment |
[production] |
| 01:10 |
<egardner@deploy1003> |
egardner: Backport for [[gerrit:1304914|Inject service RepoGroup into Hooks]], [[gerrit:1304909|MMV Beta Viewer: Improve loading/navigation UX (T429193)]], [[gerrit:1304910|Take the feature out of beta (T429509)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 01:10 |
<brett@dns7002> |
END - running authdns-update |
[production] |
| 01:08 |
<brett@dns7002> |
START - running authdns-update |
[production] |
| 00:57 |
<brett@dns7002> |
END - running authdns-update |
[production] |
| 00:56 |
<brett@dns7002> |
START - running authdns-update |
[production] |
| 00:51 |
<egardner@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1304914|Inject service RepoGroup into Hooks]], [[gerrit:1304909|MMV Beta Viewer: Improve loading/navigation UX (T429193)]], [[gerrit:1304910|Take the feature out of beta (T429509)]] |
[production] |
| 00:46 |
<swfrench-wmf> |
manually started scap-clean-images.service on deply1003 to reclaim /srv space (93% -> 66% utilization) |
[production] |
| 00:39 |
<mutante> |
codesearch10: systemctl start codesearch-write-config; systemctl restart hound-operations (gerrit:1304848) (T429819) |
[production] |
| 00:33 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1027.eqiad.wmnet with OS trixie |
[production] |
| 00:33 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:33 |
<jclark@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:30 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1033.eqiad.wmnet with OS trixie |
[production] |
| 00:30 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:29 |
<jclark@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:24 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1026.eqiad.wmnet with OS trixie |
[production] |
| 00:24 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:24 |
<jclark@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:20 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1028.eqiad.wmnet with OS trixie |
[production] |
| 00:20 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:20 |
<jclark@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:17 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1027.eqiad.wmnet with reason: host reimage |
[production] |
| 00:15 |
<egardner@deploy1003> |
sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.6,1.47.0-wmf.7,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/me |
[production] |
| 00:14 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1033.eqiad.wmnet with reason: host reimage |
[production] |
| 00:11 |
<egardner@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1304914|Inject service RepoGroup into Hooks]], [[gerrit:1304909|MMV Beta Viewer: Improve loading/navigation UX (T429193)]], [[gerrit:1304910|Take the feature out of beta (T429509)]] |
[production] |
| 00:10 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1029.eqiad.wmnet with OS trixie |
[production] |
| 00:10 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |
| 00:10 |
<jclark@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" |
[production] |