|
2026-02-27
§
|
| 05:55 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 05:51 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:49 |
<marostegui@cumin1003> |
END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1162 T418553', diff saved to https://phabricator.wikimedia.org/P89074 and previous config saved to /var/cache/conftool/dbconfig/20260227-054833-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db1222 to s2 primary T418553', diff saved to https://phabricator.wikimedia.org/P89073 and previous config saved to /var/cache/conftool/dbconfig/20260227-054750-marostegui.json |
[production] |
| 05:47 |
<marostegui> |
Starting s2 eqiad failover from db1162 to db1222 - T418553 |
[production] |
| 05:44 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T418553 |
[production] |
| 05:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db1222 with weight 0 T418553', diff saved to https://phabricator.wikimedia.org/P89072 and previous config saved to /var/cache/conftool/dbconfig/20260227-054410-marostegui.json |
[production] |
| 02:14 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 13m 18s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 01:45 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ms-fe2024.codfw.wmnet |
[production] |
| 00:01 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host ms-fe2024.codfw.wmnet |
[production] |
| 00:01 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ms-fe2024.codfw.wmnet |
[production] |
|
2026-02-26
§
|
| 23:15 |
<swfrench@deploy2002> |
Finished scap sync-world: helmfile-only deployment to clear chart version diff (duration: 02m 31s) |
[production] |
| 23:14 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2051.codfw.wmnet with OS trixie |
[production] |
| 23:13 |
<swfrench@deploy2002> |
Started scap sync-world: helmfile-only deployment to clear chart version diff |
[production] |
| 23:10 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2049.codfw.wmnet with OS trixie |
[production] |
| 23:09 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2050.codfw.wmnet with OS trixie |
[production] |
| 23:04 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2053.codfw.wmnet with OS trixie |
[production] |
| 22:58 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host ms-fe2024.codfw.wmnet |
[production] |
| 22:58 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2052.codfw.wmnet with OS trixie |
[production] |
| 22:54 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2048.codfw.wmnet with OS trixie |
[production] |
| 22:53 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2051.codfw.wmnet with reason: host reimage |
[production] |
| 22:49 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2049.codfw.wmnet with reason: host reimage |
[production] |
| 22:45 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2050.codfw.wmnet with reason: host reimage |
[production] |
| 22:41 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2053.codfw.wmnet with reason: host reimage |
[production] |
| 22:37 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2052.codfw.wmnet with reason: host reimage |
[production] |
| 22:33 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2048.codfw.wmnet with reason: host reimage |
[production] |
| 22:29 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2053.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2052.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2050.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2051.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2049.codfw.wmnet with reason: host reimage |
[production] |
| 22:27 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2048.codfw.wmnet with reason: host reimage |
[production] |
| 22:19 |
<catrope@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] (duration: 11m 48s) |
[production] |
| 22:16 |
<catrope@deploy2002> |
catrope, d3r1ck01: Continuing with sync |
[production] |
| 22:15 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2053.codfw.wmnet with OS trixie |
[production] |
| 22:14 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2052.codfw.wmnet with OS trixie |
[production] |
| 22:14 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2051.codfw.wmnet with OS trixie |
[production] |
| 22:14 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2050.codfw.wmnet with OS trixie |
[production] |
| 22:14 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2049.codfw.wmnet with OS trixie |
[production] |
| 22:13 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2048.codfw.wmnet with OS trixie |
[production] |
| 22:10 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp2043.codfw.wmnet |
[production] |
| 22:09 |
<catrope@deploy2002> |
catrope, d3r1ck01: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 22:09 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2046.codfw.wmnet with OS trixie |
[production] |
| 22:08 |
<catrope@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] |
[production] |
| 22:06 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2047.codfw.wmnet with OS trixie |
[production] |
| 22:04 |
<catrope@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1242543|Remove workaround for T370517, no longer needed (T370517)]] (duration: 07m 03s) |
[production] |