|
2026-05-26
§
|
| 06:15 |
<fceratto@dns1005> |
END - running authdns-update |
[production] |
| 06:14 |
<fceratto@dns1005> |
START - running authdns-update |
[production] |
| 06:11 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write T425622', diff saved to https://phabricator.wikimedia.org/P92909 and previous config saved to /var/cache/conftool/dbconfig/20260526-061114-fceratto.json |
[production] |
| 06:10 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T425622', diff saved to https://phabricator.wikimedia.org/P92908 and previous config saved to /var/cache/conftool/dbconfig/20260526-061021-fceratto.json |
[production] |
| 06:10 |
<federico3> |
Starting s2 eqiad failover from db1222 to db1162 - T425622 |
[production] |
| 06:04 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Set db1162 with weight 0 T425622', diff saved to https://phabricator.wikimedia.org/P92907 and previous config saved to /var/cache/conftool/dbconfig/20260526-060443-fceratto.json |
[production] |
| 06:04 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 T425622 |
[production] |
| 06:02 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) |
[production] |
| 06:02 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.global-read-only |
[production] |
| 06:01 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) |
[production] |
| 06:00 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.global-read-only |
[production] |
| 05:15 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1014.eqiad.wmnet: Maintenance on pc4 |
[production] |
| 05:15 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) |
[production] |
| 05:15 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.parsercache |
[production] |
| 05:15 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool pc1014.eqiad.wmnet: Maintenance on pc4 |
[production] |
| 05:12 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2024.codfw.wmnet,pc[1014,1024].eqiad.wmnet with reason: Maintenance on pc4 |
[production] |
| 04:37 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 04:34 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 04:02 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.47.0-wmf.1 (duration: 02m 32s) |
[production] |
| 03:39 |
<mwpresync@deploy1003> |
Finished scap sync-world: testwikis to 1.47.0-wmf.4 refs T423913 (duration: 36m 24s) |
[production] |
| 03:03 |
<mwpresync@deploy1003> |
Started scap sync-world: testwikis to 1.47.0-wmf.4 refs T423913 |
[production] |
| 02:07 |
<mwpresync@deploy1003> |
Finished scap build-images: Publishing wmf/next image (duration: 06m 20s) |
[production] |
| 02:01 |
<mwpresync@deploy1003> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2026-05-25
§
|
| 21:00 |
<jiji@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1045.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" |
[production] |
| 20:49 |
<jiji@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:38 |
<jiji@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts mc1045.eqiad.wmnet |
[production] |
| 20:37 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1044.eqiad.wmnet |
[production] |
| 20:37 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:37 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" |
[production] |
| 20:25 |
<jiji@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1044.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" |
[production] |
| 20:15 |
<moritzm> |
truncate krb5kdc.log1 (which made log rotation fail) |
[production] |
| 20:06 |
<jiji@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:57 |
<jiji@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts mc1044.eqiad.wmnet |
[production] |
| 19:25 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc1043.eqiad.wmnet |
[production] |
| 19:25 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:25 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" |
[production] |
| 19:22 |
<jiji@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc1043.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1003" |
[production] |
| 18:49 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad |
[production] |
| 18:49 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet |
[production] |
| 18:34 |
<sukhe@puppetserver1001> |
conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] |
[production] |
| 18:33 |
<sukhe@puppetserver1001> |
conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet [reason: manually pooling after reboot as icinga was down] |
[production] |
| 18:22 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5030*} and A:cp |
[production] |
| 18:22 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet |
[production] |
| 18:15 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp5023*} and A:cp |
[production] |
| 18:15 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet |
[production] |
| 18:10 |
<jiji@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:10 |
<sukhe@cumin1003> |
START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp5030*} and A:cp |
[production] |
| 18:09 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp1113*} and A:cp |
[production] |
| 18:09 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet |
[production] |
| 18:09 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet |
[production] |