|
2025-09-10
§
|
| 23:58 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1209* gradually with 4 steps - Work done |
[production] |
| 23:15 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1257* gradually with 4 steps - Work done |
[production] |
| 20:25 |
<swfrench-wmf> |
ran authdns-update to convert rest-gateway to active/passive - T400131 |
[production] |
| 18:50 |
<swfrench-wmf> |
running puppet agent on A:dnsbox hosts - T400131 |
[production] |
| 18:43 |
<swfrench-wmf> |
temporarily disabling puppet agent on A:dnsbox hosts - T400131 |
[production] |
| 18:39 |
<swfrench-wmf> |
ran authdns-update to add rest-gateway-ro and point rest-gateway at it - T400131 |
[production] |
| 18:33 |
<ladsgroup@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1256* gradually with 4 steps - Work done |
[production] |
| 18:28 |
<dduvall> |
elevated error rate during wmf.18 group1 promotion. all were `$aspect must use one of the XXX_USAGE constants` error occurring from wmf.17 (cc T404238) |
[production] |
| 18:16 |
<swfrench-wmf> |
running puppet agent on A:dnsbox hosts - T400131 |
[production] |
| 18:15 |
<dduvall@deploy1003> |
rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.18 refs T396379 |
[production] |
| 18:10 |
<btullis@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moving 4 servers to the analytics vlan - btullis@cumin1003" |
[production] |
| 18:10 |
<btullis@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moving 4 servers to the analytics vlan - btullis@cumin1003" |
[production] |
| 17:47 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1256* gradually with 4 steps - Work done |
[production] |
| 16:55 |
<swfrench-wmf> |
started single-replica PHP 8.3 pilot on shellbox-constraints - T403284 |
[production] |
| 14:57 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2212 gradually with 4 steps - pooling in |
[production] |
| 14:56 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.pool db2212 gradually with 4 steps - pooling in |
[production] |
| 14:01 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 12:51 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2212.codfw.wmnet with reason: Maintenance |
[production] |
| 12:34 |
<ladsgroup@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1236* gradually with 4 steps - Work done |
[production] |
| 12:30 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2191 gradually with 4 steps - Upgrade of db2191.codfw.wmnet completed |
[production] |
| 11:56 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1173 gradually with 4 steps - Upgrade of db1173.eqiad.wmnet completed |
[production] |
| 11:48 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1236* gradually with 4 steps - Work done |
[production] |
| 11:44 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.pool db2191 gradually with 4 steps - Upgrade of db2191.codfw.wmnet completed |
[production] |
| 11:10 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.pool db1173 gradually with 4 steps - Upgrade of db1173.eqiad.wmnet completed |
[production] |
| 01:12 |
<mwpresync@deploy1003> |
Finished scap build-images: Publishing wmf/next image (duration: 12m 08s) |
[production] |
| 01:00 |
<mwpresync@deploy1003> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2025-09-09
§
|
| 18:39 |
<dduvall@deploy1003> |
rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.18 refs T396379 |
[production] |
| 17:38 |
<swfrench-wmf> |
migrated shellbox-syntaxhighlight to PHP 8.3 in eqiad - T403284 |
[production] |
| 17:15 |
<swfrench-wmf> |
migrated shellbox-syntaxhighlight to PHP 8.3 in codfw - T403284 |
[production] |
| 17:12 |
<swfrench-wmf> |
restored locally modified `helmfile.d/dse-k8s-services/_airflow_common_/values-dev.yaml` duplicating https://gerrit.wikimedia.org/r/1186497 on deploy1003 to unstick deployment-charts updates |
[production] |
| 16:48 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2240.codfw.wmnet with reason: Maintenance |
[production] |
| 16:48 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2239.codfw.wmnet with reason: Maintenance |
[production] |
| 16:00 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2237.codfw.wmnet with reason: Maintenance |
[production] |
| 15:12 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2236.codfw.wmnet with reason: Maintenance |
[production] |
| 14:43 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1193 gradually with 4 steps - Pool db1193.eqiad.wmnet in after cloning |
[production] |
| 14:24 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2219.codfw.wmnet with reason: Maintenance |
[production] |
| 14:15 |
<ladsgroup@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1181* gradually with 4 steps - Work done |
[production] |
| 13:58 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.pool db1193 gradually with 4 steps - Pool db1193.eqiad.wmnet in after cloning |
[production] |
| 13:36 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2210.codfw.wmnet with reason: Maintenance |
[production] |
| 13:29 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1181* gradually with 4 steps - Work done |
[production] |
| 12:58 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1172 gradually with 4 steps - Pool db1172.eqiad.wmnet in after cloning |
[production] |
| 12:48 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2206.codfw.wmnet with reason: Maintenance |
[production] |
| 12:48 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2199.codfw.wmnet with reason: Maintenance |
[production] |
| 12:12 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.pool db1172 gradually with 4 steps - Pool db1172.eqiad.wmnet in after cloning |
[production] |