|
2026-02-27
§
|
| 06:00 |
<marostegui> |
revert: Failover m5-master T401966 |
[production] |
| 06:00 |
<marostegui> |
Starting s4 eqiad failover from db1244 to db1160 - T418079 |
[production] |
| 05:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T418079 |
[production] |
| 05:58 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 05:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db1160 with weight 0 T418079', diff saved to https://phabricator.wikimedia.org/P89077 and previous config saved to /var/cache/conftool/dbconfig/20260227-055845-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2151 (T418465)', diff saved to https://phabricator.wikimedia.org/P89076 and previous config saved to /var/cache/conftool/dbconfig/20260227-055835-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2151.codfw.wmnet with reason: Maintenance |
[production] |
| 05:57 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 05:56 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 05:55 |
<marostegui> |
Failover m5-master T401966 |
[production] |
| 05:55 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 05:51 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:49 |
<marostegui@cumin1003> |
END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1162 T418553', diff saved to https://phabricator.wikimedia.org/P89074 and previous config saved to /var/cache/conftool/dbconfig/20260227-054833-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db1222 to s2 primary T418553', diff saved to https://phabricator.wikimedia.org/P89073 and previous config saved to /var/cache/conftool/dbconfig/20260227-054750-marostegui.json |
[production] |
| 05:47 |
<marostegui> |
Starting s2 eqiad failover from db1162 to db1222 - T418553 |
[production] |
| 05:44 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T418553 |
[production] |
| 05:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db1222 with weight 0 T418553', diff saved to https://phabricator.wikimedia.org/P89072 and previous config saved to /var/cache/conftool/dbconfig/20260227-054410-marostegui.json |
[production] |
| 02:14 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 13m 18s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 01:45 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ms-fe2024.codfw.wmnet |
[production] |
| 01:18 |
<wm-bot2> |
Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894) |
[tools.cluebotng-review] |
| 00:01 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host ms-fe2024.codfw.wmnet |
[production] |
| 00:01 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ms-fe2024.codfw.wmnet |
[production] |
|
2026-02-26
§
|
| 23:15 |
<swfrench@deploy2002> |
Finished scap sync-world: helmfile-only deployment to clear chart version diff (duration: 02m 31s) |
[production] |
| 23:14 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2051.codfw.wmnet with OS trixie |
[production] |
| 23:13 |
<swfrench@deploy2002> |
Started scap sync-world: helmfile-only deployment to clear chart version diff |
[production] |
| 23:10 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2049.codfw.wmnet with OS trixie |
[production] |
| 23:09 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2050.codfw.wmnet with OS trixie |
[production] |
| 23:04 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2053.codfw.wmnet with OS trixie |
[production] |
| 22:58 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host ms-fe2024.codfw.wmnet |
[production] |
| 22:58 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2052.codfw.wmnet with OS trixie |
[production] |
| 22:54 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2048.codfw.wmnet with OS trixie |
[production] |
| 22:53 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2051.codfw.wmnet with reason: host reimage |
[production] |
| 22:49 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2049.codfw.wmnet with reason: host reimage |
[production] |
| 22:45 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2050.codfw.wmnet with reason: host reimage |
[production] |
| 22:41 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2053.codfw.wmnet with reason: host reimage |
[production] |
| 22:37 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2052.codfw.wmnet with reason: host reimage |
[production] |
| 22:36 |
<eileen> |
civicrm upgraded from 6995dc03 to 162bbf7c |
[fundraising] |
| 22:33 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2048.codfw.wmnet with reason: host reimage |
[production] |
| 22:29 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2053.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2052.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2050.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2051.codfw.wmnet with reason: host reimage |
[production] |
| 22:28 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2049.codfw.wmnet with reason: host reimage |
[production] |
| 22:27 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2048.codfw.wmnet with reason: host reimage |
[production] |
| 22:19 |
<catrope@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] (duration: 11m 48s) |
[production] |
| 22:16 |
<catrope@deploy2002> |
catrope, d3r1ck01: Continuing with sync |
[production] |