|
2022-11-24
ยง
|
| 08:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323214)', diff saved to https://phabricator.wikimedia.org/P40951 and previous config saved to /var/cache/conftool/dbconfig/20221124-082458-ladsgroup.json |
[production] |
| 08:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P40950 and previous config saved to /var/cache/conftool/dbconfig/20221124-082447-marostegui.json |
[production] |
| 08:13 |
<moritzm> |
installing tomcat9 security updates |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P40949 and previous config saved to /var/cache/conftool/dbconfig/20221124-080941-marostegui.json |
[production] |
| 08:04 |
<moritzm> |
rebalance Ganeti group A/codfw following reboots |
[production] |
| 07:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40948 and previous config saved to /var/cache/conftool/dbconfig/20221124-075434-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40947 and previous config saved to /var/cache/conftool/dbconfig/20221124-075226-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40946 and previous config saved to /var/cache/conftool/dbconfig/20221124-075205-marostegui.json |
[production] |
| 07:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323214)', diff saved to https://phabricator.wikimedia.org/P40945 and previous config saved to /var/cache/conftool/dbconfig/20221124-074517-ladsgroup.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P40944 and previous config saved to /var/cache/conftool/dbconfig/20221124-073658-marostegui.json |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1201 (T323214)', diff saved to https://phabricator.wikimedia.org/P40943 and previous config saved to /var/cache/conftool/dbconfig/20221124-073637-ladsgroup.json |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T323214)', diff saved to https://phabricator.wikimedia.org/P40942 and previous config saved to /var/cache/conftool/dbconfig/20221124-073616-ladsgroup.json |
[production] |
| 07:30 |
<phedenskog@deploy1002> |
Finished deploy [performance/navtiming@e421904]: (no justification provided) (duration: 00m 08s) |
[production] |
| 07:30 |
<phedenskog@deploy1002> |
Started deploy [performance/navtiming@e421904]: (no justification provided) |
[production] |
| 07:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P40941 and previous config saved to /var/cache/conftool/dbconfig/20221124-073011-ladsgroup.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P40940 and previous config saved to /var/cache/conftool/dbconfig/20221124-072152-marostegui.json |
[production] |
| 07:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P40939 and previous config saved to /var/cache/conftool/dbconfig/20221124-072110-ladsgroup.json |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P40938 and previous config saved to /var/cache/conftool/dbconfig/20221124-071504-ladsgroup.json |
[production] |
| 07:14 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:14 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:09 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/citoid: apply |
[production] |
| 07:09 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 07:08 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 07:07 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 07:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40936 and previous config saved to /var/cache/conftool/dbconfig/20221124-070645-marostegui.json |
[production] |
| 07:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P40935 and previous config saved to /var/cache/conftool/dbconfig/20221124-070603-ladsgroup.json |
[production] |
| 07:05 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 07:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depool db1181 T323117', diff saved to https://phabricator.wikimedia.org/P40934 and previous config saved to /var/cache/conftool/dbconfig/20221124-070546-ladsgroup.json |
[production] |
| 07:05 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 07:05 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 07:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1099:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40933 and previous config saved to /var/cache/conftool/dbconfig/20221124-070437-marostegui.json |
[production] |
| 07:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:03 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db1136 to s7 primary and set section read-write T323117', diff saved to https://phabricator.wikimedia.org/P40932 and previous config saved to /var/cache/conftool/dbconfig/20221124-070250-ladsgroup.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set s7 eqiad as read-only for maintenance - T323117', diff saved to https://phabricator.wikimedia.org/P40931 and previous config saved to /var/cache/conftool/dbconfig/20221124-070215-ladsgroup.json |
[production] |
| 07:02 |
<Amir1> |
Starting s7 eqiad failover from db1181 to db1136 - T323117 |
[production] |
| 06:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323214)', diff saved to https://phabricator.wikimedia.org/P40930 and previous config saved to /var/cache/conftool/dbconfig/20221124-065956-ladsgroup.json |
[production] |
| 06:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2118.codfw.wmnet with reason: Maintenance |
[production] |