|
2022-04-21
ยง
|
| 20:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26046 and previous config saved to /var/cache/conftool/dbconfig/20220421-201817-ladsgroup.json |
[production] |
| 20:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26045 and previous config saved to /var/cache/conftool/dbconfig/20220421-201659-ladsgroup.json |
[production] |
| 20:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26044 and previous config saved to /var/cache/conftool/dbconfig/20220421-201455-ladsgroup.json |
[production] |
| 20:14 |
<mutante> |
reimaging gitlab-runner2001.codfw.wmnet one more time to confirm things work from scratch now T297659 |
[production] |
| 20:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26043 and previous config saved to /var/cache/conftool/dbconfig/20220421-201313-ladsgroup.json |
[production] |
| 20:09 |
<mutante> |
[ganeti2021:~] $ sudo gnt-instance shutdown gitlab-runner2001.codfw.wmnet |
[production] |
| 20:08 |
<mutante> |
[puppetmaster1001:~] $ sudo puppet cert clean gitlab-runner2001.codfw.wmnet |
[production] |
| 20:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26042 and previous config saved to /var/cache/conftool/dbconfig/20220421-200312-ladsgroup.json |
[production] |
| 20:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26041 and previous config saved to /var/cache/conftool/dbconfig/20220421-200154-ladsgroup.json |
[production] |
| 19:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26040 and previous config saved to /var/cache/conftool/dbconfig/20220421-195950-ladsgroup.json |
[production] |
| 19:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26039 and previous config saved to /var/cache/conftool/dbconfig/20220421-195808-ladsgroup.json |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26038 and previous config saved to /var/cache/conftool/dbconfig/20220421-194807-ladsgroup.json |
[production] |
| 19:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1136 (T298565)', diff saved to https://phabricator.wikimedia.org/P26037 and previous config saved to /var/cache/conftool/dbconfig/20220421-194419-ladsgroup.json |
[production] |
| 19:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P26036 and previous config saved to /var/cache/conftool/dbconfig/20220421-194303-ladsgroup.json |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26035 and previous config saved to /var/cache/conftool/dbconfig/20220421-193302-ladsgroup.json |
[production] |
| 19:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1165 (T306560)', diff saved to https://phabricator.wikimedia.org/P26034 and previous config saved to /var/cache/conftool/dbconfig/20220421-193052-ladsgroup.json |
[production] |
| 19:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26033 and previous config saved to /var/cache/conftool/dbconfig/20220421-193039-ladsgroup.json |
[production] |
| 19:23 |
<cdanis> |
depooling & disabling puppet on cp2029 for some manual testing T303534 |
[production] |
| 19:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P26032 and previous config saved to /var/cache/conftool/dbconfig/20220421-192330-ladsgroup.json |
[production] |
| 19:23 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:23 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26031 and previous config saved to /var/cache/conftool/dbconfig/20220421-192322-ladsgroup.json |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P26030 and previous config saved to /var/cache/conftool/dbconfig/20220421-191847-ladsgroup.json |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26029 and previous config saved to /var/cache/conftool/dbconfig/20220421-191534-ladsgroup.json |
[production] |
| 19:08 |
<ebernhardson> |
set index.unassigned.node_left.delayed_timeout to null for all indices on elasticsearch-eqiad-psi (:9200), reverting previous test of 10m back to defaults |
[production] |
| 19:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26028 and previous config saved to /var/cache/conftool/dbconfig/20220421-190817-ladsgroup.json |
[production] |
| 19:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26027 and previous config saved to /var/cache/conftool/dbconfig/20220421-190029-ladsgroup.json |
[production] |
| 18:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26026 and previous config saved to /var/cache/conftool/dbconfig/20220421-185312-ladsgroup.json |
[production] |
| 18:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T306560)', diff saved to https://phabricator.wikimedia.org/P26025 and previous config saved to /var/cache/conftool/dbconfig/20220421-184523-ladsgroup.json |
[production] |
| 18:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26024 and previous config saved to /var/cache/conftool/dbconfig/20220421-183807-ladsgroup.json |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P26023 and previous config saved to /var/cache/conftool/dbconfig/20220421-181614-ladsgroup.json |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P26022 and previous config saved to /var/cache/conftool/dbconfig/20220421-181601-ladsgroup.json |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |