|
2022-06-15
ยง
|
| 10:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T310011)', diff saved to https://phabricator.wikimedia.org/P29778 and previous config saved to /var/cache/conftool/dbconfig/20220615-103048-marostegui.json |
[production] |
| 10:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1029 es1030 es1028 for kernel upgrade', diff saved to https://phabricator.wikimedia.org/P29777 and previous config saved to /var/cache/conftool/dbconfig/20220615-102929-root.json |
[production] |
| 10:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P29776 and previous config saved to /var/cache/conftool/dbconfig/20220615-101543-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29775 and previous config saved to /var/cache/conftool/dbconfig/20220615-100235-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P29774 and previous config saved to /var/cache/conftool/dbconfig/20220615-100037-marostegui.json |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4001.ulsfo.wmnet |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T310011)', diff saved to https://phabricator.wikimedia.org/P29773 and previous config saved to /var/cache/conftool/dbconfig/20220615-094532-marostegui.json |
[production] |
| 09:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti4001.ulsfo.wmnet |
[production] |
| 09:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29772 and previous config saved to /var/cache/conftool/dbconfig/20220615-092706-marostegui.json |
[production] |
| 09:20 |
<marostegui> |
Reboot sanitarium hosts (db1154, db1155) wiki replicas will have lag |
[production] |
| 09:14 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reimage for host ms-be1059.eqiad.wmnet with OS bullseye |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1119 (T310011)', diff saved to https://phabricator.wikimedia.org/P29771 and previous config saved to /var/cache/conftool/dbconfig/20220615-091257-marostegui.json |
[production] |
| 09:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1119.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1119.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29770 and previous config saved to /var/cache/conftool/dbconfig/20220615-091249-marostegui.json |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P29769 and previous config saved to /var/cache/conftool/dbconfig/20220615-091201-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P29768 and previous config saved to /var/cache/conftool/dbconfig/20220615-085744-marostegui.json |
[production] |
| 08:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P29767 and previous config saved to /var/cache/conftool/dbconfig/20220615-085656-marostegui.json |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P29766 and previous config saved to /var/cache/conftool/dbconfig/20220615-084239-marostegui.json |
[production] |
| 08:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29765 and previous config saved to /var/cache/conftool/dbconfig/20220615-084151-marostegui.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29764 and previous config saved to /var/cache/conftool/dbconfig/20220615-084046-marostegui.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P29763 and previous config saved to /var/cache/conftool/dbconfig/20220615-083554-root.json |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29762 and previous config saved to /var/cache/conftool/dbconfig/20220615-082734-marostegui.json |
[production] |
| 08:23 |
<jnuche@deploy1002> |
Installation of scap version "4.9.3" completed for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installing scap version "4.9.3" for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installation of scap version "4.9.3" completed for 557 hosts |
[production] |
| 08:22 |
<jnuche@deploy1002> |
Installing scap version "4.9.3" for 557 hosts |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P29761 and previous config saved to /var/cache/conftool/dbconfig/20220615-082050-root.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P29760 and previous config saved to /var/cache/conftool/dbconfig/20220615-081744-root.json |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P29759 and previous config saved to /var/cache/conftool/dbconfig/20220615-080546-root.json |
[production] |
| 08:03 |
<XioNoX> |
re-enable BGP to Telia in eqsin for optic replacement - T300485 |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P29758 and previous config saved to /var/cache/conftool/dbconfig/20220615-080240-root.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1148 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P29757 and previous config saved to /var/cache/conftool/dbconfig/20220615-075042-root.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1099:3311 (T310011)', diff saved to https://phabricator.wikimedia.org/P29756 and previous config saved to /var/cache/conftool/dbconfig/20220615-075024-marostegui.json |
[production] |
| 07:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |