|
2022-06-15
ยง
|
| 08:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P29763 and previous config saved to /var/cache/conftool/dbconfig/20220615-083554-root.json |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29762 and previous config saved to /var/cache/conftool/dbconfig/20220615-082734-marostegui.json |
[production] |
| 08:23 |
<jnuche@deploy1002> |
Installation of scap version "4.9.3" completed for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installing scap version "4.9.3" for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installation of scap version "4.9.3" completed for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installing scap version "4.9.3" for 557 hosts |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P29761 and previous config saved to /var/cache/conftool/dbconfig/20220615-082050-root.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P29760 and previous config saved to /var/cache/conftool/dbconfig/20220615-081744-root.json |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P29759 and previous config saved to /var/cache/conftool/dbconfig/20220615-080546-root.json |
[production] |
| 08:03 |
<XioNoX> |
re-enable BGP to Telia in eqsin for optic replacement - T300485 |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P29758 and previous config saved to /var/cache/conftool/dbconfig/20220615-080240-root.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P29757 and previous config saved to /var/cache/conftool/dbconfig/20220615-075042-root.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1099:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29756 and previous config saved to /var/cache/conftool/dbconfig/20220615-075024-marostegui.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:50 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P29755 and previous config saved to /var/cache/conftool/dbconfig/20220615-074736-root.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P29754 and previous config saved to /var/cache/conftool/dbconfig/20220615-073538-root.json |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P29753 and previous config saved to /var/cache/conftool/dbconfig/20220615-073232-root.json |
[production] |
| 07:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29752 and previous config saved to /var/cache/conftool/dbconfig/20220615-072352-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P29751 and previous config saved to /var/cache/conftool/dbconfig/20220615-072034-root.json |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P29750 and previous config saved to /var/cache/conftool/dbconfig/20220615-071728-root.json |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P29749 and previous config saved to /var/cache/conftool/dbconfig/20220615-070847-marostegui.json |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P29748 and previous config saved to /var/cache/conftool/dbconfig/20220615-065342-marostegui.json |
[production] |
| 06:52 |
<XioNoX> |
disable BGP to Telia in eqsin for optic replacement - T300485 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29747 and previous config saved to /var/cache/conftool/dbconfig/20220615-063837-marostegui.json |
[production] |
| 06:02 |
<marostegui> |
Reboot db[2071-2078] T310485 |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29746 and previous config saved to /var/cache/conftool/dbconfig/20220615-060153-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29745 and previous config saved to /var/cache/conftool/dbconfig/20220615-054252-marostegui.json |
[production] |
| 05:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:42 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1173.eqiad.wmnet with OS bullseye |
[production] |
| 05:17 |
<marostegui> |
dbmaint es5@codfw T310485 |
[production] |
| 05:17 |
<marostegui> |
dbmaint es4@codfw T310485 |
[production] |
| 05:17 |
<marostegui> |
dbmaint es3@codfw T310485 |
[production] |
| 05:17 |
<marostegui> |
dbmaint es2@codfw T310485 |
[production] |
| 05:17 |
<marostegui> |
dbmaint es1@codfw T310485 |
[production] |
| 05:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage |
[production] |
| 05:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage |
[production] |
| 05:03 |
<marostegui> |
Reboot dbproxy1016 and dbproxy1021 T310484 |
[production] |
| 04:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1173.eqiad.wmnet with OS bullseye |
[production] |
| 02:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |