|
2023-08-22
ยง
|
| 06:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1212 (T344589)', diff saved to https://phabricator.wikimedia.org/P50781 and previous config saved to /var/cache/conftool/dbconfig/20230822-060246-ladsgroup.json |
[production] |
| 06:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T343718)', diff saved to https://phabricator.wikimedia.org/P50780 and previous config saved to /var/cache/conftool/dbconfig/20230822-060147-ladsgroup.json |
[production] |
| 06:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write T344621', diff saved to https://phabricator.wikimedia.org/P50779 and previous config saved to /var/cache/conftool/dbconfig/20230822-060131-ladsgroup.json |
[production] |
| 06:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T344621', diff saved to https://phabricator.wikimedia.org/P50778 and previous config saved to /var/cache/conftool/dbconfig/20230822-060104-ladsgroup.json |
[production] |
| 06:00 |
<Amir1> |
Starting s1 eqiad failover from db1163 to db1184 - T344621 |
[production] |
| 05:58 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 12897 |
[production] |
| 05:57 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1114.eqiad.wmnet with reason: host reimage |
[production] |
| 05:56 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 12897 |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T344589)', diff saved to https://phabricator.wikimedia.org/P50777 and previous config saved to /var/cache/conftool/dbconfig/20230822-055548-ladsgroup.json |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1212 (T344589)', diff saved to https://phabricator.wikimedia.org/P50776 and previous config saved to /var/cache/conftool/dbconfig/20230822-055528-ladsgroup.json |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198 (T344589)', diff saved to https://phabricator.wikimedia.org/P50775 and previous config saved to /var/cache/conftool/dbconfig/20230822-055457-ladsgroup.json |
[production] |
| 05:54 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1113.eqiad.wmnet with reason: host reimage |
[production] |
| 05:53 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1114.eqiad.wmnet with reason: host reimage |
[production] |
| 05:51 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1113.eqiad.wmnet with reason: host reimage |
[production] |
| 05:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2137:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50774 and previous config saved to /var/cache/conftool/dbconfig/20230822-055101-ladsgroup.json |
[production] |
| 05:51 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T343718)', diff saved to https://phabricator.wikimedia.org/P50773 and previous config saved to /var/cache/conftool/dbconfig/20230822-055050-ladsgroup.json |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1185 (T343718)', diff saved to https://phabricator.wikimedia.org/P50772 and previous config saved to /var/cache/conftool/dbconfig/20230822-055040-ladsgroup.json |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T343718)', diff saved to https://phabricator.wikimedia.org/P50771 and previous config saved to /var/cache/conftool/dbconfig/20230822-055030-ladsgroup.json |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2177 (T344589)', diff saved to https://phabricator.wikimedia.org/P50770 and previous config saved to /var/cache/conftool/dbconfig/20230822-054920-ladsgroup.json |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 05:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T344589)', diff saved to https://phabricator.wikimedia.org/P50769 and previous config saved to /var/cache/conftool/dbconfig/20230822-054855-ladsgroup.json |
[production] |
| 05:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P50768 and previous config saved to /var/cache/conftool/dbconfig/20230822-053951-ladsgroup.json |
[production] |
| 05:38 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1114.eqiad.wmnet with OS bullseye |
[production] |
| 05:37 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1113.eqiad.wmnet with OS bullseye |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P50767 and previous config saved to /var/cache/conftool/dbconfig/20230822-053544-ladsgroup.json |
[production] |
| 05:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P50766 and previous config saved to /var/cache/conftool/dbconfig/20230822-053524-ladsgroup.json |
[production] |
| 05:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P50765 and previous config saved to /var/cache/conftool/dbconfig/20230822-053349-ladsgroup.json |
[production] |
| 05:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P50764 and previous config saved to /var/cache/conftool/dbconfig/20230822-052445-ladsgroup.json |
[production] |
| 05:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P50763 and previous config saved to /var/cache/conftool/dbconfig/20230822-052038-ladsgroup.json |
[production] |
| 05:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P50762 and previous config saved to /var/cache/conftool/dbconfig/20230822-052018-ladsgroup.json |
[production] |
| 05:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P50761 and previous config saved to /var/cache/conftool/dbconfig/20230822-051843-ladsgroup.json |
[production] |
| 05:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set db1184 with weight 0 T344621', diff saved to https://phabricator.wikimedia.org/P50760 and previous config saved to /var/cache/conftool/dbconfig/20230822-051347-ladsgroup.json |
[production] |
| 05:13 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344621 |
[production] |
| 05:13 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T344621 |
[production] |
| 05:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T344589)', diff saved to https://phabricator.wikimedia.org/P50759 and previous config saved to /var/cache/conftool/dbconfig/20230822-051204-ladsgroup.json |
[production] |
| 05:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198 (T344589)', diff saved to https://phabricator.wikimedia.org/P50758 and previous config saved to /var/cache/conftool/dbconfig/20230822-050938-ladsgroup.json |
[production] |
| 05:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1184 (T344589)', diff saved to https://phabricator.wikimedia.org/P50757 and previous config saved to /var/cache/conftool/dbconfig/20230822-050543-ladsgroup.json |
[production] |
| 05:05 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T343718)', diff saved to https://phabricator.wikimedia.org/P50756 and previous config saved to /var/cache/conftool/dbconfig/20230822-050532-ladsgroup.json |
[production] |
| 05:05 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T343718)', diff saved to https://phabricator.wikimedia.org/P50755 and previous config saved to /var/cache/conftool/dbconfig/20230822-050511-ladsgroup.json |
[production] |