|
2024-07-01
ยง
|
| 10:39 |
<fabfur@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs |
[production] |
| 10:39 |
<fabfur@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs |
[production] |
| 10:38 |
<elukey@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:37 |
<elukey@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 10:37 |
<elukey@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:37 |
<elukey@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2130 (T364069)', diff saved to https://phabricator.wikimedia.org/P65580 and previous config saved to /var/cache/conftool/dbconfig/20240701-102633-marostegui.json |
[production] |
| 10:26 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 10:26 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 10:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T364069)', diff saved to https://phabricator.wikimedia.org/P65579 and previous config saved to /var/cache/conftool/dbconfig/20240701-102611-marostegui.json |
[production] |
| 10:23 |
<fabfur> |
upgrading A:cp-drmrs to haproxy 2.8.10 (T367756) |
[production] |
| 10:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P65578 and previous config saved to /var/cache/conftool/dbconfig/20240701-101104-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P65577 and previous config saved to /var/cache/conftool/dbconfig/20240701-095557-marostegui.json |
[production] |
| 09:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P65576 and previous config saved to /var/cache/conftool/dbconfig/20240701-094547-root.json |
[production] |
| 09:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P65575 and previous config saved to /var/cache/conftool/dbconfig/20240701-094341-root.json |
[production] |
| 09:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T364069)', diff saved to https://phabricator.wikimedia.org/P65574 and previous config saved to /var/cache/conftool/dbconfig/20240701-094050-marostegui.json |
[production] |
| 09:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P65573 and previous config saved to /var/cache/conftool/dbconfig/20240701-093042-root.json |
[production] |
| 09:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P65572 and previous config saved to /var/cache/conftool/dbconfig/20240701-092835-root.json |
[production] |
| 09:26 |
<btullis@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:15 |
<btullis@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P65570 and previous config saved to /var/cache/conftool/dbconfig/20240701-091536-root.json |
[production] |
| 09:14 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1051060|JsonSchemaValidator: Measure duration (T365245)]] (duration: 22m 15s) |
[production] |
| 09:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P65569 and previous config saved to /var/cache/conftool/dbconfig/20240701-091329-root.json |
[production] |
| 09:06 |
<urbanecm@deploy1002> |
urbanecm: Continuing with sync |
[production] |
| 09:06 |
<urbanecm@deploy1002> |
urbanecm: Backport for [[gerrit:1051060|JsonSchemaValidator: Measure duration (T365245)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 09:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P65568 and previous config saved to /var/cache/conftool/dbconfig/20240701-090031-root.json |
[production] |
| 08:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P65567 and previous config saved to /var/cache/conftool/dbconfig/20240701-085824-root.json |
[production] |
| 08:51 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:1051060|JsonSchemaValidator: Measure duration (T365245)]] |
[production] |
| 08:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P65566 and previous config saved to /var/cache/conftool/dbconfig/20240701-084525-root.json |
[production] |
| 08:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P65565 and previous config saved to /var/cache/conftool/dbconfig/20240701-084318-root.json |
[production] |
| 08:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P65564 and previous config saved to /var/cache/conftool/dbconfig/20240701-083020-root.json |
[production] |
| 08:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P65563 and previous config saved to /var/cache/conftool/dbconfig/20240701-082813-root.json |
[production] |
| 08:18 |
<jynus@cumin1002> |
dbctl commit (dc=all): 'Depool es1025 for backups T363812', diff saved to https://phabricator.wikimedia.org/P65562 and previous config saved to /var/cache/conftool/dbconfig/20240701-081811-jynus.json |
[production] |
| 08:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P65561 and previous config saved to /var/cache/conftool/dbconfig/20240701-081514-root.json |
[production] |
| 08:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1195 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P65560 and previous config saved to /var/cache/conftool/dbconfig/20240701-081307-root.json |
[production] |
| 08:07 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1169.eqiad.wmnet onto db1195.eqiad.wmnet |
[production] |
| 07:44 |
<elukey> |
`apt-get clean` on buil2001 to free some space in the root partition |
[production] |
| 07:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Place db1195 in s1 T368871', diff saved to https://phabricator.wikimedia.org/P65559 and previous config saved to /var/cache/conftool/dbconfig/20240701-070243-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.clone of db1169.eqiad.wmnet onto db1195.eqiad.wmnet |
[production] |
| 06:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1169 T368871', diff saved to https://phabricator.wikimedia.org/P65558 and previous config saved to /var/cache/conftool/dbconfig/20240701-063601-root.json |
[production] |
| 06:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2116 (T364069)', diff saved to https://phabricator.wikimedia.org/P65557 and previous config saved to /var/cache/conftool/dbconfig/20240701-063344-marostegui.json |
[production] |
| 06:33 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 06:33 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 05:02 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: Reboot |
[production] |
| 05:02 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: Reboot |
[production] |
| 04:56 |
<marostegui> |
Failover m2 from db1195 to db1228 - T368494 |
[production] |
| 04:52 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2133,2160].codfw.wmnet,db[1195,1217,1228].eqiad.wmnet with reason: m2 switchover T368494 |
[production] |
| 04:51 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db[2133,2160].codfw.wmnet,db[1195,1217,1228].eqiad.wmnet with reason: m2 switchover T368494 |
[production] |
| 04:50 |
<marostegui> |
dbmaint eqiad Rebuild pagelinks table on s8 master T364069 |
[production] |
| 04:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1156 (T367856)', diff saved to https://phabricator.wikimedia.org/P65556 and previous config saved to /var/cache/conftool/dbconfig/20240701-044945-marostegui.json |
[production] |