|
2021-11-03
ยง
|
| 14:37 |
<elukey@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 14:33 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'staging' . |
[production] |
| 14:32 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 14:31 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 14:31 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 14:30 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:21 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 14:20 |
<hnowlan> |
rolling restart-php7.2-fpm on A:mw-eqiad and A:mw-api-eqiad |
[production] |
| 14:17 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 14:16 |
<hnowlan> |
deploying wikidiff2-1.13.0-1 to A:mw-eqiad and A:mw-api-eqiad |
[production] |
| 14:13 |
<moritzm> |
installing remaining tiff security updates for buster |
[production] |
| 14:10 |
<moritzm> |
initialising ganeti-test01.svc.codfw.wmnet cluster on ganeti-test2001 T286206 |
[production] |
| 14:07 |
<XioNoX> |
move cr2-codfw access switches link to working linecard - T289241 |
[production] |
| 14:04 |
<vgutierrez> |
update eqsin and ulsfo cp instances to ATS 8.0.8-1wm5 - T294897 |
[production] |
| 13:38 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 13:34 |
<bblack@cumin1001> |
conftool action : set/pooled=no; selector: name=cp403[3456].*,service=ats-be |
[production] |
| 13:34 |
<bblack> |
cp403[3456] - depool ats-be service (upcoming re-reimage) |
[production] |
| 12:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:21 |
<vgutierrez> |
update trafficserver on cp4027 to 8.0.8-1wm5 - T294897 |
[production] |
| 12:20 |
<vgutierrez> |
update trafficserver on cp4021 to 8.0.8-1wm5 - T294897 |
[production] |
| 12:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:18 |
<vgutierrez> |
upload trafficserver 8.0.8-1wm5 to apt.wm.org (buster) - T294897 |
[production] |
| 12:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:15 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 9ca753bf4b7afea41c29225d4f32e3ba01bf7c30: Revert "Adjust AF config for ukwiki" (T272330) (duration: 01m 03s) |
[production] |
| 12:13 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 667ef0b6e9e8d1d70061cc904ce49e7632300b75: foundationwiki: Increase AF throttle requirements (duration: 01m 13s) |
[production] |
| 11:58 |
<hnowlan> |
rolling restart-php7.2-fpm on A:mw-codfw and A:mw-api-codfw |
[production] |
| 11:56 |
<hnowlan> |
deploying wikidiff2-1.13.0-1 to A:mw-codfw and A:mw-api-codfw |
[production] |
| 11:37 |
<Amir1> |
start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https |
[production] |
| 11:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:14 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 7fdf3f5476d9d8ab45eb793090613e328a91bb7a: Wikisource: allow copy-uploads from Commons (T294824) (duration: 01m 04s) |
[production] |
| 11:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:23 |
<XioNoX> |
re-enable eqiad Equinix IXP peerings - T290877 |
[production] |
| 08:55 |
<XioNoX> |
Disable eqiad Equinix IXP peerings - T290877 |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove logpager replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17660 and previous config saved to /var/cache/conftool/dbconfig/20211103-075801-marostegui.json |
[production] |
| 07:58 |
<elukey@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 07:57 |
<elukey@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 07:50 |
<marostegui> |
Drop oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer from foundationwiki T294595 |
[production] |
| 06:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS buster |
[production] |
| 06:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 06:35 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 06:35 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 34888b034e54ec35ca3b6745336fc0881e50c9b0: Growth IP research survey: Fix coverage (T294568) (duration: 01m 04s) |
[production] |
| 06:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS buster |
[production] |
| 06:10 |
<marostegui> |
Stop replication on db1163 T290865 |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1163 until it's reimaged to buster T293964', diff saved to https://phabricator.wikimedia.org/P17659 and previous config saved to /var/cache/conftool/dbconfig/20211103-060644-root.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1118 to s1 primary and set section read-write T293964', diff saved to https://phabricator.wikimedia.org/P17658 and previous config saved to /var/cache/conftool/dbconfig/20211103-060201-root.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T293964', diff saved to https://phabricator.wikimedia.org/P17657 and previous config saved to /var/cache/conftool/dbconfig/20211103-060114-root.json |
[production] |
| 06:00 |
<marostegui> |
Starting s1 eqiad failover from db1163 to db1118 - T293964 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 |
[production] |