|
2022-01-25
ยง
|
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19104 and previous config saved to /var/cache/conftool/dbconfig/20220125-085228-root.json |
[production] |
| 08:45 |
<moritzm> |
draining instances off ganeti1005 for reimage |
[production] |
| 08:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti1013.eqiad.wmnet with OS buster |
[production] |
| 08:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19103 and previous config saved to /var/cache/conftool/dbconfig/20220125-083724-root.json |
[production] |
| 08:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 08:32 |
<jayme> |
kubernetes staging migrated tainted worker node setup - T290967 |
[production] |
| 08:32 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestagemaster1001.eqiad.wmnet |
[production] |
| 08:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 08:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 08:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 08:25 |
<marostegui@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Revert: Promote pc1013 to master in pc3 T299046 (duration: 00m 49s) |
[production] |
| 08:25 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubestagemaster1001.eqiad.wmnet |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T285149)', diff saved to https://phabricator.wikimedia.org/P19102 and previous config saved to /var/cache/conftool/dbconfig/20220125-082326-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T285149)', diff saved to https://phabricator.wikimedia.org/P19101 and previous config saved to /var/cache/conftool/dbconfig/20220125-082319-marostegui.json |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19100 and previous config saved to /var/cache/conftool/dbconfig/20220125-082220-root.json |
[production] |
| 08:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19099 and previous config saved to /var/cache/conftool/dbconfig/20220125-080814-marostegui.json |
[production] |
| 08:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19098 and previous config saved to /var/cache/conftool/dbconfig/20220125-080717-root.json |
[production] |
| 07:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19097 and previous config saved to /var/cache/conftool/dbconfig/20220125-075309-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 40%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19096 and previous config saved to /var/cache/conftool/dbconfig/20220125-075213-root.json |
[production] |
| 07:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T285149)', diff saved to https://phabricator.wikimedia.org/P19095 and previous config saved to /var/cache/conftool/dbconfig/20220125-073805-marostegui.json |
[production] |
| 07:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19094 and previous config saved to /var/cache/conftool/dbconfig/20220125-073709-root.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T285149)', diff saved to https://phabricator.wikimedia.org/P19093 and previous config saved to /var/cache/conftool/dbconfig/20220125-073457-marostegui.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T285149)', diff saved to https://phabricator.wikimedia.org/P19092 and previous config saved to /var/cache/conftool/dbconfig/20220125-073450-marostegui.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 20%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19091 and previous config saved to /var/cache/conftool/dbconfig/20220125-072206-root.json |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P19090 and previous config saved to /var/cache/conftool/dbconfig/20220125-071945-marostegui.json |
[production] |
| 07:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1013.eqiad.wmnet with OS bullseye |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 10%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19089 and previous config saved to /var/cache/conftool/dbconfig/20220125-070702-root.json |
[production] |
| 07:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P19088 and previous config saved to /var/cache/conftool/dbconfig/20220125-070441-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 5%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19087 and previous config saved to /var/cache/conftool/dbconfig/20220125-065158-root.json |
[production] |
| 06:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T285149)', diff saved to https://phabricator.wikimedia.org/P19086 and previous config saved to /var/cache/conftool/dbconfig/20220125-064936-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc1013.eqiad.wmnet with OS bullseye |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T285149)', diff saved to https://phabricator.wikimedia.org/P19085 and previous config saved to /var/cache/conftool/dbconfig/20220125-064829-marostegui.json |
[production] |
| 06:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T285149)', diff saved to https://phabricator.wikimedia.org/P19084 and previous config saved to /var/cache/conftool/dbconfig/20220125-064801-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 1%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19083 and previous config saved to /var/cache/conftool/dbconfig/20220125-063655-root.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1030.eqiad.wmnet with OS bullseye |
[production] |
| 06:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 06:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 06:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P19082 and previous config saved to /var/cache/conftool/dbconfig/20220125-063256-marostegui.json |
[production] |
| 06:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |