|
2023-11-29
ยง
|
| 10:50 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 10:49 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 10:37 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cp[1075-1090].eqiad.wmnet |
[production] |
| 10:37 |
<btullis> |
pausing all active dags on all airflow instances |
[production] |
| 10:36 |
<fabfur> |
decommissioning cp1075-1090 (T352253) |
[production] |
| 10:10 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:10 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1025.eqiad.wmnet with OS bookworm |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53938 and previous config saved to /var/cache/conftool/dbconfig/20231129-092808-root.json |
[production] |
| 09:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1025.eqiad.wmnet with reason: host reimage |
[production] |
| 09:20 |
<hashar@deploy2002> |
Synchronized php: group1 wikis to 1.42.0-wmf.7 refs T350083 (duration: 07m 23s) |
[production] |
| 09:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1025.eqiad.wmnet with reason: host reimage |
[production] |
| 09:13 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.7 refs T350083 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53937 and previous config saved to /var/cache/conftool/dbconfig/20231129-091303-root.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbproxy1025.eqiad.wmnet with OS bookworm |
[production] |
| 09:02 |
<hashar@deploy2002> |
Finished scap: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] (duration: 09m 39s) |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53936 and previous config saved to /var/cache/conftool/dbconfig/20231129-085758-root.json |
[production] |
| 08:55 |
<hashar@deploy2002> |
hashar and anzx: Continuing with sync |
[production] |
| 08:54 |
<marostegui> |
Failover m1-master from dbproxy1022 to dbproxy1024 T351864 |
[production] |
| 08:53 |
<hashar@deploy2002> |
hashar and anzx: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:52 |
<hashar@deploy2002> |
Started scap: Backport for [[gerrit:975379|zghwiki: add logos (T350241)]] |
[production] |
| 08:48 |
<hashar@deploy2002> |
Finished scap: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] (duration: 10m 10s) |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53935 and previous config saved to /var/cache/conftool/dbconfig/20231129-084253-root.json |
[production] |
| 08:41 |
<hashar@deploy2002> |
hashar and anzx: Continuing with sync |
[production] |
| 08:39 |
<hashar@deploy2002> |
hashar and anzx: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:38 |
<hashar@deploy2002> |
Started scap: Backport for [[gerrit:978023|Enable VisualEditor in the Appendix namespace on enwiktionary (T350926)]] |
[production] |
| 08:33 |
<marostegui> |
Drop oathauth_users from centralauth T348693 |
[production] |
| 08:28 |
<marostegui> |
oathauth_users from fishbowl.dblist T348693 |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53934 and previous config saved to /var/cache/conftool/dbconfig/20231129-082748-root.json |
[production] |
| 08:22 |
<marostegui> |
oathauth_users from private.dblist T348693 |
[production] |
| 08:19 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] (duration: 08m 01s) |
[production] |
| 08:13 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 08:12 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53933 and previous config saved to /var/cache/conftool/dbconfig/20231129-081243-root.json |
[production] |
| 08:11 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] |
[production] |
| 08:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1012.eqiad.wmnet with OS bookworm |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53932 and previous config saved to /var/cache/conftool/dbconfig/20231129-075738-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1027.eqiad.wmnet with OS bookworm |
[production] |
| 07:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1012.eqiad.wmnet with reason: host reimage |
[production] |
| 07:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1027.eqiad.wmnet with reason: host reimage |
[production] |
| 07:38 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc1012.eqiad.wmnet with reason: host reimage |
[production] |
| 07:37 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on es1027.eqiad.wmnet with reason: host reimage |
[production] |
| 07:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc1012.eqiad.wmnet with OS bookworm |
[production] |
| 07:25 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] (duration: 09m 25s) |
[production] |
| 07:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host es1027.eqiad.wmnet with OS bookworm |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1027 T351916', diff saved to https://phabricator.wikimedia.org/P53931 and previous config saved to /var/cache/conftool/dbconfig/20231129-072306-root.json |
[production] |
| 07:18 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 07:17 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:15 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] |
[production] |
| 07:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Switch |
[production] |