|
2022-03-30
ยง
|
| 02:34 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:34 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance |
[production] |
| 02:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance |
[production] |
| 02:33 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 02:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 02:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23673 and previous config saved to /var/cache/conftool/dbconfig/20220330-023344-ladsgroup.json |
[production] |
| 02:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P23672 and previous config saved to /var/cache/conftool/dbconfig/20220330-021839-ladsgroup.json |
[production] |
| 02:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1106 (T300775)', diff saved to https://phabricator.wikimedia.org/P23671 and previous config saved to /var/cache/conftool/dbconfig/20220330-021111-marostegui.json |
[production] |
| 02:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T300775)', diff saved to https://phabricator.wikimedia.org/P23670 and previous config saved to /var/cache/conftool/dbconfig/20220330-021058-marostegui.json |
[production] |
| 02:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P23669 and previous config saved to /var/cache/conftool/dbconfig/20220330-020334-ladsgroup.json |
[production] |
| 01:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P23668 and previous config saved to /var/cache/conftool/dbconfig/20220330-015552-marostegui.json |
[production] |
| 01:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1127 (T298565)', diff saved to https://phabricator.wikimedia.org/P23667 and previous config saved to /var/cache/conftool/dbconfig/20220330-015527-ladsgroup.json |
[production] |
| 01:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23666 and previous config saved to /var/cache/conftool/dbconfig/20220330-015519-ladsgroup.json |
[production] |
| 01:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23665 and previous config saved to /var/cache/conftool/dbconfig/20220330-014829-ladsgroup.json |
[production] |
| 01:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23664 and previous config saved to /var/cache/conftool/dbconfig/20220330-014621-ladsgroup.json |
[production] |
| 01:46 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:46 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23663 and previous config saved to /var/cache/conftool/dbconfig/20220330-014549-ladsgroup.json |
[production] |
| 01:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P23662 and previous config saved to /var/cache/conftool/dbconfig/20220330-014047-marostegui.json |
[production] |
| 01:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23661 and previous config saved to /var/cache/conftool/dbconfig/20220330-014014-ladsgroup.json |
[production] |
| 01:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P23660 and previous config saved to /var/cache/conftool/dbconfig/20220330-013044-ladsgroup.json |
[production] |
| 01:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T300775)', diff saved to https://phabricator.wikimedia.org/P23659 and previous config saved to /var/cache/conftool/dbconfig/20220330-012542-marostegui.json |
[production] |
| 01:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23658 and previous config saved to /var/cache/conftool/dbconfig/20220330-012509-ladsgroup.json |
[production] |
| 01:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P23657 and previous config saved to /var/cache/conftool/dbconfig/20220330-011539-ladsgroup.json |
[production] |
| 01:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23656 and previous config saved to /var/cache/conftool/dbconfig/20220330-011004-ladsgroup.json |
[production] |
| 01:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23655 and previous config saved to /var/cache/conftool/dbconfig/20220330-010034-ladsgroup.json |
[production] |
| 00:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 00:38 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 00:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 00:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 00:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23654 and previous config saved to /var/cache/conftool/dbconfig/20220330-002523-ladsgroup.json |
[production] |
| 00:25 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:25 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298565)', diff saved to https://phabricator.wikimedia.org/P23653 and previous config saved to /var/cache/conftool/dbconfig/20220330-002515-ladsgroup.json |
[production] |
| 00:24 |
<catrope@deploy1002> |
Finished scap: Update Kashmiri namespace names (T304790) (duration: 12m 29s) |
[production] |
| 00:12 |
<catrope@deploy1002> |
Started scap: Update Kashmiri namespace names (T304790) |
[production] |
| 00:12 |
<catrope@deploy1002> |
scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. (duration: 00m 28s) |
[production] |
| 00:11 |
<catrope@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 00:11 |
<catrope@deploy1002> |
Started scap: Update Kashmiri namespace names (T304790) |
[production] |
| 00:10 |
<catrope@deploy1002> |
scap failed: RuntimeError Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. (duration: 00m 28s) |
[production] |
| 00:10 |
<catrope@deploy1002> |
Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 00:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P23652 and previous config saved to /var/cache/conftool/dbconfig/20220330-001010-ladsgroup.json |
[production] |
| 00:09 |
<catrope@deploy1002> |
Started scap: Update Kashmiri namespace names (T304790) |
[production] |
| 00:07 |
<catrope@deploy1002> |
scap failed: RuntimeError Scap failed!: 6/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. (duration: 04m 32s) |
[production] |