|
2026-09-24
ยง
|
| 12:59 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:59 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:58 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:58 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:52 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:52 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:43 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:43 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply |
[production] |
| 12:34 |
<urbanecm@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply |
[production] |
| 12:34 |
<urbanecm@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-experimental: apply |
[production] |
| 12:04 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:03 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 11:21 |
<vgutierrez@cumin1004> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828) |
[production] |
| 11:13 |
<hnowlan> |
restarted restbase on restbase2029 |
[production] |
| 11:04 |
<vgutierrez@cumin1004> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828) |
[production] |
| 10:50 |
<hnowlan> |
deleting stuck mw-web pods in eqiad |
[production] |
| 10:45 |
<dreamyjazz@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) |
[production] |
| 10:44 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply |
[production] |
| 10:42 |
<vgutierrez@cumin1004> |
END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828) |
[production] |
| 10:40 |
<dreamyjazz@deploy1003> |
dreamyjazz: Continuing with deployment |
[production] |
| 10:39 |
<dreamyjazz@deploy1003> |
dreamyjazz: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 10:36 |
<filippo@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision |
[production] |
| 10:35 |
<dreamyjazz@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] |
[production] |
| 10:34 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply |
[production] |
| 10:32 |
<dreamyjazz@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) |
[production] |
| 10:29 |
<trueg@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply |
[production] |
| 10:26 |
<dreamyjazz@deploy1003> |
dreamyjazz: Continuing with deployment |
[production] |
| 10:26 |
<trueg@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply |
[production] |
| 10:25 |
<dreamyjazz@deploy1003> |
dreamyjazz: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 10:23 |
<filippo@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision |
[production] |
| 10:22 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply |
[production] |
| 10:21 |
<dreamyjazz@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] |
[production] |
| 10:17 |
<rzl@deploy1003> |
Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) |
[production] |
| 10:17 |
<hnowlan@deploy1003> |
Forcefully removing global lock: Unlocking scap after restoration of power in codfw |
[production] |
| 10:12 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply |
[production] |
| 10:11 |
<vgutierrez@cumin1004> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828) |
[production] |
| 10:08 |
<vgutierrez@cumin1004> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828) |
[production] |
| 10:03 |
<moritzm> |
installing apr-util security updates |
[production] |
| 09:46 |
<moritzm> |
installing bind9 security updates (client-side tools/libs only) |
[production] |
| 09:40 |
<vgutierrez@puppetserver1001> |
conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet |
[production] |
| 09:27 |
<ayounsi@cumin1004> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, T437984] |
[production] |
| 09:27 |
<ayounsi@cumin1004> |
START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, T437984] |
[production] |
| 09:26 |
<ayounsi@cumin1004> |
END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 |
[production] |
| 09:25 |
<ayounsi@cumin1004> |
START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 |
[production] |
| 09:23 |
<arthurtaylor@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:22 |
<arthurtaylor@deploy1003> |
helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:22 |
<arthurtaylor@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:22 |
<arthurtaylor@deploy1003> |
helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:21 |
<arthurtaylor@deploy1003> |
helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:21 |
<arthurtaylor@deploy1003> |
helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply |
[production] |