|
2022-08-30
ยง
|
| 06:19 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2153 (T316186)', diff saved to https://phabricator.wikimedia.org/P33654 and previous config saved to /var/cache/conftool/dbconfig/20220830-061926-ladsgroup.json |
[production] |
| 06:19 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 06:19 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 06:19 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T316186)', diff saved to https://phabricator.wikimedia.org/P33653 and previous config saved to /var/cache/conftool/dbconfig/20220830-061901-ladsgroup.json |
[production] |
| 06:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2176 (T316186)', diff saved to https://phabricator.wikimedia.org/P33652 and previous config saved to /var/cache/conftool/dbconfig/20220830-061243-ladsgroup.json |
[production] |
| 06:12 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 06:12 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance |
[production] |
| 06:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112 (T316186)', diff saved to https://phabricator.wikimedia.org/P33651 and previous config saved to /var/cache/conftool/dbconfig/20220830-061218-ladsgroup.json |
[production] |
| 06:07 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:07 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2112 (T316186)', diff saved to https://phabricator.wikimedia.org/P33650 and previous config saved to /var/cache/conftool/dbconfig/20220830-060554-ladsgroup.json |
[production] |
| 06:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depool db1173 T316110 T312984 T312863 T316186', diff saved to https://phabricator.wikimedia.org/P33649 and previous config saved to /var/cache/conftool/dbconfig/20220830-060543-ladsgroup.json |
[production] |
| 06:05 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 06:05 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: Maintenance |
[production] |
| 06:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T316186)', diff saved to https://phabricator.wikimedia.org/P33648 and previous config saved to /var/cache/conftool/dbconfig/20220830-060509-ladsgroup.json |
[production] |
| 06:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db1131 to s6 primary and set section read-write T316110', diff saved to https://phabricator.wikimedia.org/P33647 and previous config saved to /var/cache/conftool/dbconfig/20220830-060109-ladsgroup.json |
[production] |
| 06:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set s6 eqiad as read-only for maintenance - T316110', diff saved to https://phabricator.wikimedia.org/P33646 and previous config saved to /var/cache/conftool/dbconfig/20220830-060026-ladsgroup.json |
[production] |
| 06:00 |
<Amir1> |
Starting s6 eqiad failover from db1173 to db1131 - T316110 |
[production] |
| 05:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2116 (T316186)', diff saved to https://phabricator.wikimedia.org/P33645 and previous config saved to /var/cache/conftool/dbconfig/20220830-055948-ladsgroup.json |
[production] |
| 05:59 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 05:59 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 05:56 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance |
[production] |
| 05:56 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130 (T316186)', diff saved to https://phabricator.wikimedia.org/P33644 and previous config saved to /var/cache/conftool/dbconfig/20220830-055555-ladsgroup.json |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2130 (T316186)', diff saved to https://phabricator.wikimedia.org/P33643 and previous config saved to /var/cache/conftool/dbconfig/20220830-054924-ladsgroup.json |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2174 (T316186)', diff saved to https://phabricator.wikimedia.org/P33642 and previous config saved to /var/cache/conftool/dbconfig/20220830-054859-ladsgroup.json |
[production] |
| 05:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2174 (T316186)', diff saved to https://phabricator.wikimedia.org/P33641 and previous config saved to /var/cache/conftool/dbconfig/20220830-054242-ladsgroup.json |
[production] |
| 05:42 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 05:42 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance |
[production] |
| 05:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T316186)', diff saved to https://phabricator.wikimedia.org/P33640 and previous config saved to /var/cache/conftool/dbconfig/20220830-054217-ladsgroup.json |
[production] |
| 05:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2173 (T316186)', diff saved to https://phabricator.wikimedia.org/P33639 and previous config saved to /var/cache/conftool/dbconfig/20220830-053559-ladsgroup.json |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T316186)', diff saved to https://phabricator.wikimedia.org/P33638 and previous config saved to /var/cache/conftool/dbconfig/20220830-053529-ladsgroup.json |
[production] |
| 05:33 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2146 (T316186)', diff saved to https://phabricator.wikimedia.org/P33637 and previous config saved to /var/cache/conftool/dbconfig/20220830-052930-ladsgroup.json |
[production] |
| 05:29 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2146.codfw.wmnet with reason: Maintenance |
[production] |
| 05:29 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2146.codfw.wmnet with reason: Maintenance |
[production] |
| 05:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set db1131 with weight 0 T316110', diff saved to https://phabricator.wikimedia.org/P33636 and previous config saved to /var/cache/conftool/dbconfig/20220830-051106-ladsgroup.json |
[production] |
| 05:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s6 T316110 |
[production] |
| 05:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s6 T316110 |
[production] |
| 05:03 |
<ryankemper> |
T306899 T316496 Deployed WCQS `0.3.115`. That should (hopefully) resolve these tickets. |
[production] |
| 05:01 |
<ryankemper> |
[WCQS Deploy] Restarted `wcqs-updater` across all hosts: `sudo -E cumin 'A:wcqs-public' 'systemctl restart wcqs-updater'` |
[production] |