|
2023-11-29
ยง
|
| 11:14 |
<hnowlan@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 11:13 |
<hnowlan@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 11:13 |
<hnowlan@deploy2002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 11:12 |
<btullis> |
re-enabled all DAGs on all airflow instances after airflow upgrade to 2.7.3 |
[production] |
| 10:57 |
<vgutierrez> |
upload ipip-multiqueue-optimizer 0.3+deb11u1 to apt.wm.o (bullseye) - T352249 |
[production] |
| 10:56 |
<jiji@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply |
[production] |
| 10:56 |
<jiji@deploy2002> |
helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply |
[production] |
| 10:53 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 10:51 |
<hnowlan@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 10:51 |
<hnowlan@deploy2002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 10:50 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 10:49 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 10:37 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cp[1075-1090].eqiad.wmnet |
[production] |
| 10:37 |
<btullis> |
pausing all active dags on all airflow instances |
[production] |
| 10:36 |
<fabfur> |
decommissioning cp1075-1090 (T352253) |
[production] |
| 10:10 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:10 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1025.eqiad.wmnet with OS bookworm |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53938 and previous config saved to /var/cache/conftool/dbconfig/20231129-092808-root.json |
[production] |
| 09:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1025.eqiad.wmnet with reason: host reimage |
[production] |
| 09:20 |
<hashar@deploy2002> |
Synchronized php: group1 wikis to 1.42.0-wmf.7 refs T350083 (duration: 07m 23s) |
[production] |
| 09:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1025.eqiad.wmnet with reason: host reimage |
[production] |
| 09:13 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.7 refs T350083 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53937 and previous config saved to /var/cache/conftool/dbconfig/20231129-091303-root.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbproxy1025.eqiad.wmnet with OS bookworm |
[production] |
| 09:02 |
<hashar@deploy2002> |
Finished scap: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] (duration: 09m 39s) |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53936 and previous config saved to /var/cache/conftool/dbconfig/20231129-085758-root.json |
[production] |
| 08:55 |
<hashar@deploy2002> |
hashar and anzx: Continuing with sync |
[production] |
| 08:54 |
<marostegui> |
Failover m1-master from dbproxy1022 to dbproxy1024 T351864 |
[production] |
| 08:53 |
<hashar@deploy2002> |
hashar and anzx: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:52 |
<hashar@deploy2002> |
Started scap: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] |
[production] |
| 08:48 |
<hashar@deploy2002> |
Finished scap: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] (duration: 10m 10s) |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53935 and previous config saved to /var/cache/conftool/dbconfig/20231129-084253-root.json |
[production] |
| 08:41 |
<hashar@deploy2002> |
hashar and anzx: Continuing with sync |
[production] |
| 08:39 |
<hashar@deploy2002> |
hashar and anzx: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:38 |
<hashar@deploy2002> |
Started scap: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] |
[production] |
| 08:33 |
<marostegui> |
Drop oathauth_users from centralauth T348693 |
[production] |
| 08:28 |
<marostegui> |
oathauth_users from fishbowl.dblist T348693 |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53934 and previous config saved to /var/cache/conftool/dbconfig/20231129-082748-root.json |
[production] |
| 08:22 |
<marostegui> |
oathauth_users from private.dblist T348693 |
[production] |
| 08:19 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] (duration: 08m 01s) |
[production] |
| 08:13 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 08:12 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53933 and previous config saved to /var/cache/conftool/dbconfig/20231129-081243-root.json |
[production] |
| 08:11 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] |
[production] |
| 08:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1012.eqiad.wmnet with OS bookworm |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53932 and previous config saved to /var/cache/conftool/dbconfig/20231129-075738-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1027.eqiad.wmnet with OS bookworm |
[production] |
| 07:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1012.eqiad.wmnet with reason: host reimage |
[production] |
| 07:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1027.eqiad.wmnet with reason: host reimage |
[production] |