|
2022-03-29
ยง
|
| 22:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23635 and previous config saved to /var/cache/conftool/dbconfig/20220329-222128-ladsgroup.json |
[production] |
| 22:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P23634 and previous config saved to /var/cache/conftool/dbconfig/20220329-221634-ladsgroup.json |
[production] |
| 22:14 |
<mutante> |
doc1001 - rebooting (doc.wikimedia.org) |
[production] |
| 22:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P23633 and previous config saved to /var/cache/conftool/dbconfig/20220329-220623-ladsgroup.json |
[production] |
| 22:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23632 and previous config saved to /var/cache/conftool/dbconfig/20220329-220128-ladsgroup.json |
[production] |
| 21:54 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 21:54 |
<mutante> |
cumin1001 systemctl start httpbb_hourly_appserver |
[production] |
| 21:54 |
<mutante> |
cumin1001 systemctl status httpbb_hourly_appserver |
[production] |
| 21:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P23631 and previous config saved to /var/cache/conftool/dbconfig/20220329-215118-ladsgroup.json |
[production] |
| 21:48 |
<mutante> |
doc1002 - rebooting |
[production] |
| 21:46 |
<mutante> |
doc2001 - rebooting |
[production] |
| 21:38 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 21:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23630 and previous config saved to /var/cache/conftool/dbconfig/20220329-213613-ladsgroup.json |
[production] |
| 21:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23629 and previous config saved to /var/cache/conftool/dbconfig/20220329-212804-ladsgroup.json |
[production] |
| 21:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23628 and previous config saved to /var/cache/conftool/dbconfig/20220329-212756-ladsgroup.json |
[production] |
| 21:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:22 |
<mutante> |
aphlict1001 - manually starting aphlict service after reboot (was needed for some reason) |
[production] |
| 21:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:19 |
<ryankemper@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wdqs2007.codfw.wmnet |
[production] |
| 21:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:17 |
<mutante> |
aphlict1001 - rebooting - this will temp break Phabricator realtime notifications but will be back shortly |
[production] |
| 21:17 |
<mutante> |
planet1002 - rebooting |
[production] |
| 21:14 |
<mutante> |
planet2002 - rebooting |
[production] |
| 21:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P23627 and previous config saved to /var/cache/conftool/dbconfig/20220329-211251-ladsgroup.json |
[production] |
| 21:10 |
<mutante> |
phab1004 - rebooting |
[production] |
| 21:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23626 and previous config saved to /var/cache/conftool/dbconfig/20220329-210856-ladsgroup.json |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23625 and previous config saved to /var/cache/conftool/dbconfig/20220329-210848-ladsgroup.json |
[production] |
| 21:05 |
<mutante> |
phab2002 - rebooting |
[production] |
| 21:04 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 20:59 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773320|Fix I7ce58529cdd320a9500dc215291ef1c369cee9d3: Rearranging restriction levels and add editautopatrolprotected for eliminators. (T303579)]] (duration: 00m 56s) |
[production] |
| 20:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P23624 and previous config saved to /var/cache/conftool/dbconfig/20220329-205746-ladsgroup.json |
[production] |
| 20:57 |
<catrope@deploy1002> |
Synchronized php-1.39.0-wmf.5/skins/Vector/skin.json: Backport: [[gerrit:774840|Restore the classes skin-vector and skin-vector-search-vue to body]] (duration: 00m 55s) |
[production] |
| 20:54 |
<catrope@deploy1002> |
Synchronized php-1.39.0-wmf.4/skins/Vector: Backport: Revert: [[gerrit:774839|End migration mode]] (duration: 00m 53s) |
[production] |
| 20:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P23623 and previous config saved to /var/cache/conftool/dbconfig/20220329-205343-ladsgroup.json |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:50 |
<catrope@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 20:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |