|
2022-05-31
ยง
|
| 15:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1100 (re)pooling @ 5%: Maint done', diff saved to https://phabricator.wikimedia.org/P29221 and previous config saved to /var/cache/conftool/dbconfig/20220531-153859-ladsgroup.json |
[production] |
| 15:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T307525)', diff saved to https://phabricator.wikimedia.org/P29220 and previous config saved to /var/cache/conftool/dbconfig/20220531-153846-ladsgroup.json |
[production] |
| 15:34 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P29219 and previous config saved to /var/cache/conftool/dbconfig/20220531-153422-ladsgroup.json |
[production] |
| 15:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1136 (T309311)', diff saved to https://phabricator.wikimedia.org/P29218 and previous config saved to /var/cache/conftool/dbconfig/20220531-153053-ladsgroup.json |
[production] |
| 15:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29217 and previous config saved to /var/cache/conftool/dbconfig/20220531-152341-ladsgroup.json |
[production] |
| 15:19 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P29216 and previous config saved to /var/cache/conftool/dbconfig/20220531-151916-ladsgroup.json |
[production] |
| 15:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1136 (T309311)', diff saved to https://phabricator.wikimedia.org/P29215 and previous config saved to /var/cache/conftool/dbconfig/20220531-151515-ladsgroup.json |
[production] |
| 15:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:12 |
<jelto> |
migrate gitlab-replica to gitlab1003 |
[production] |
| 15:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29214 and previous config saved to /var/cache/conftool/dbconfig/20220531-150836-ladsgroup.json |
[production] |
| 15:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T60674)', diff saved to https://phabricator.wikimedia.org/P29213 and previous config saved to /var/cache/conftool/dbconfig/20220531-150411-ladsgroup.json |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3318 (T60674)', diff saved to https://phabricator.wikimedia.org/P29211 and previous config saved to /var/cache/conftool/dbconfig/20220531-143720-ladsgroup.json |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1166 (T307525)', diff saved to https://phabricator.wikimedia.org/P29210 and previous config saved to /var/cache/conftool/dbconfig/20220531-143716-ladsgroup.json |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1109 (T60674)', diff saved to https://phabricator.wikimedia.org/P29209 and previous config saved to /var/cache/conftool/dbconfig/20220531-143712-ladsgroup.json |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:37 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T309311)', diff saved to https://phabricator.wikimedia.org/P29208 and previous config saved to /var/cache/conftool/dbconfig/20220531-143621-ladsgroup.json |
[production] |
| 14:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T307525)', diff saved to https://phabricator.wikimedia.org/P29207 and previous config saved to /var/cache/conftool/dbconfig/20220531-143538-ladsgroup.json |
[production] |
| 14:25 |
<jbond@deploy1002> |
Finished deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts (duration: 00m 52s) |
[production] |
| 14:24 |
<jbond@deploy1002> |
Started deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts |
[production] |
| 14:24 |
<jbond@deploy1002> |
Finished deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts (duration: 01m 01s) |
[production] |
| 14:23 |
<jbond@deploy1002> |
Started deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts |
[production] |
| 14:23 |
<jbond@deploy1002> |
Finished deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts (duration: 01m 52s) |
[production] |
| 14:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P29206 and previous config saved to /var/cache/conftool/dbconfig/20220531-142207-ladsgroup.json |
[production] |
| 14:21 |
<jbond@deploy1002> |
Started deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts |
[production] |
| 14:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P29205 and previous config saved to /var/cache/conftool/dbconfig/20220531-142116-ladsgroup.json |
[production] |
| 14:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29204 and previous config saved to /var/cache/conftool/dbconfig/20220531-142033-ladsgroup.json |
[production] |
| 14:20 |
<jbond@deploy1002> |
Finished deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts (duration: 00m 32s) |
[production] |
| 14:20 |
<jbond@deploy1002> |
Started deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts |
[production] |
| 14:19 |
<jbond@deploy1002> |
Finished deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts (duration: 02m 15s) |
[production] |
| 14:19 |
<tgr> |
doing an emergency revert for T309616 |
[production] |
| 14:18 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1006.wikimedia.org with OS bullseye |
[production] |
| 14:17 |
<jbond@deploy1002> |
Started deploy [netbox/deploy@7bbf659]: deploying v2.10.4-wmf6 to new hosts |
[production] |
| 14:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P29202 and previous config saved to /var/cache/conftool/dbconfig/20220531-140702-ladsgroup.json |
[production] |
| 14:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P29201 and previous config saved to /var/cache/conftool/dbconfig/20220531-140611-ladsgroup.json |
[production] |
| 14:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29200 and previous config saved to /var/cache/conftool/dbconfig/20220531-140528-ladsgroup.json |
[production] |
| 14:03 |
<jbond@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=netbox,name=eqiad |
[production] |
| 14:03 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |
| 13:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1109 (T60674)', diff saved to https://phabricator.wikimedia.org/P29199 and previous config saved to /var/cache/conftool/dbconfig/20220531-135157-ladsgroup.json |
[production] |
| 13:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T309311)', diff saved to https://phabricator.wikimedia.org/P29198 and previous config saved to /var/cache/conftool/dbconfig/20220531-135105-ladsgroup.json |
[production] |
| 13:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T307525)', diff saved to https://phabricator.wikimedia.org/P29197 and previous config saved to /var/cache/conftool/dbconfig/20220531-135022-ladsgroup.json |
[production] |
| 13:38 |
<elukey> |
move ml-etcd100[1-3] from drdb to plain to investigate high k8s latencies for the control plane |
[production] |
| 13:35 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1004.wikimedia.org with OS bullseye |
[production] |