|
2022-03-29
ยง
|
| 21:10 |
<mutante> |
phab1004 - rebooting |
[production] |
| 21:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23626 and previous config saved to /var/cache/conftool/dbconfig/20220329-210856-ladsgroup.json |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23625 and previous config saved to /var/cache/conftool/dbconfig/20220329-210848-ladsgroup.json |
[production] |
| 21:05 |
<mutante> |
phab2002 - rebooting |
[production] |
| 21:04 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 20:59 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773320|Fix I7ce58529cdd320a9500dc215291ef1c369cee9d3: Rearranging restriction levels and add editautopatrolprotected for eliminators. (T303579)]] (duration: 00m 56s) |
[production] |
| 20:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P23624 and previous config saved to /var/cache/conftool/dbconfig/20220329-205746-ladsgroup.json |
[production] |
| 20:57 |
<catrope@deploy1002> |
Synchronized php-1.39.0-wmf.5/skins/Vector/skin.json: Backport: [[gerrit:774840|Restore the classes skin-vector and skin-vector-search-vue to body]] (duration: 00m 55s) |
[production] |
| 20:54 |
<catrope@deploy1002> |
Synchronized php-1.39.0-wmf.4/skins/Vector: Backport: Revert: [[gerrit:774839|End migration mode]] (duration: 00m 53s) |
[production] |
| 20:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P23623 and previous config saved to /var/cache/conftool/dbconfig/20220329-205343-ladsgroup.json |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:50 |
<catrope@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 20:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23622 and previous config saved to /var/cache/conftool/dbconfig/20220329-204241-ladsgroup.json |
[production] |
| 20:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23621 and previous config saved to /var/cache/conftool/dbconfig/20220329-204034-ladsgroup.json |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23620 and previous config saved to /var/cache/conftool/dbconfig/20220329-204021-ladsgroup.json |
[production] |
| 20:39 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773302|Add wikimedia.com to wgNoFollowDomainExceptions (T304555)]] (duration: 01m 06s) |
[production] |
| 20:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P23619 and previous config saved to /var/cache/conftool/dbconfig/20220329-203838-ladsgroup.json |
[production] |
| 20:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:36 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:774953|[config]: Deploy gdi-safety-survey to ES,EN,FR and PT wikis]] (duration: 00m 56s) |
[production] |
| 20:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773896|Config for new android schemas]] (duration: 01m 00s) |
[production] |
| 20:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23618 and previous config saved to /var/cache/conftool/dbconfig/20220329-202516-ladsgroup.json |
[production] |
| 20:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23617 and previous config saved to /var/cache/conftool/dbconfig/20220329-202333-ladsgroup.json |
[production] |
| 20:20 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 20:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131 (T297189)', diff saved to https://phabricator.wikimedia.org/P23616 and previous config saved to /var/cache/conftool/dbconfig/20220329-201611-marostegui.json |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23615 and previous config saved to /var/cache/conftool/dbconfig/20220329-201041-ladsgroup.json |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23614 and previous config saved to /var/cache/conftool/dbconfig/20220329-201011-ladsgroup.json |
[production] |
| 20:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P23613 and previous config saved to /var/cache/conftool/dbconfig/20220329-200106-marostegui.json |
[production] |
| 19:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23612 and previous config saved to /var/cache/conftool/dbconfig/20220329-195505-ladsgroup.json |
[production] |