|
2023-08-22
ยง
|
| 08:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P50838 and previous config saved to /var/cache/conftool/dbconfig/20230822-084104-ladsgroup.json |
[production] |
| 08:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1136 (T344589)', diff saved to https://phabricator.wikimedia.org/P50837 and previous config saved to /var/cache/conftool/dbconfig/20230822-083912-ladsgroup.json |
[production] |
| 08:39 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:38 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T344589)', diff saved to https://phabricator.wikimedia.org/P50836 and previous config saved to /var/cache/conftool/dbconfig/20230822-083848-ladsgroup.json |
[production] |
| 08:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P50835 and previous config saved to /var/cache/conftool/dbconfig/20230822-083207-ladsgroup.json |
[production] |
| 08:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P50834 and previous config saved to /var/cache/conftool/dbconfig/20230822-083132-ladsgroup.json |
[production] |
| 08:31 |
<urbanecm> |
mwmaint1002: Stop frwiki instance of T315510 scripts due to a large volume of T343859 errors |
[production] |
| 08:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2021', diff saved to https://phabricator.wikimedia.org/P50833 and previous config saved to /var/cache/conftool/dbconfig/20230822-083044-ladsgroup.json |
[production] |
| 08:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P50832 and previous config saved to /var/cache/conftool/dbconfig/20230822-083004-ladsgroup.json |
[production] |
| 08:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P50831 and previous config saved to /var/cache/conftool/dbconfig/20230822-082559-ladsgroup.json |
[production] |
| 08:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P50830 and previous config saved to /var/cache/conftool/dbconfig/20230822-082342-ladsgroup.json |
[production] |
| 08:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P50829 and previous config saved to /var/cache/conftool/dbconfig/20230822-081701-ladsgroup.json |
[production] |
| 08:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P50828 and previous config saved to /var/cache/conftool/dbconfig/20230822-081626-ladsgroup.json |
[production] |
| 08:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50827 and previous config saved to /var/cache/conftool/dbconfig/20230822-081537-ladsgroup.json |
[production] |
| 08:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50826 and previous config saved to /var/cache/conftool/dbconfig/20230822-081458-ladsgroup.json |
[production] |
| 08:12 |
<moritzm> |
bounce ferm on aux-k8s-ctrl1001 |
[production] |
| 08:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P50825 and previous config saved to /var/cache/conftool/dbconfig/20230822-081054-ladsgroup.json |
[production] |
| 08:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P50824 and previous config saved to /var/cache/conftool/dbconfig/20230822-080836-ladsgroup.json |
[production] |
| 08:05 |
<moritzm> |
installing Linux 4.19.289-2 on Buster hosts |
[production] |
| 08:05 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1117.eqiad.wmnet with reason: host reimage |
[production] |
| 08:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling es2021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50823 and previous config saved to /var/cache/conftool/dbconfig/20230822-080413-ladsgroup.json |
[production] |
| 08:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling es1021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50822 and previous config saved to /var/cache/conftool/dbconfig/20230822-080328-ladsgroup.json |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1021.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1021.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:02 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1117.eqiad.wmnet with reason: host reimage |
[production] |
| 08:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210 (T343718)', diff saved to https://phabricator.wikimedia.org/P50821 and previous config saved to /var/cache/conftool/dbconfig/20230822-080155-ladsgroup.json |
[production] |
| 08:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T344589)', diff saved to https://phabricator.wikimedia.org/P50820 and previous config saved to /var/cache/conftool/dbconfig/20230822-080119-ladsgroup.json |
[production] |
| 07:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2171:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50819 and previous config saved to /var/cache/conftool/dbconfig/20230822-075728-ladsgroup.json |
[production] |
| 07:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 07:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 07:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T343718)', diff saved to https://phabricator.wikimedia.org/P50818 and previous config saved to /var/cache/conftool/dbconfig/20230822-075717-ladsgroup.json |
[production] |
| 07:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2108 (T344589)', diff saved to https://phabricator.wikimedia.org/P50817 and previous config saved to /var/cache/conftool/dbconfig/20230822-075453-ladsgroup.json |
[production] |
| 07:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 07:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 07:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T344589)', diff saved to https://phabricator.wikimedia.org/P50816 and previous config saved to /var/cache/conftool/dbconfig/20230822-075329-ladsgroup.json |
[production] |
| 07:51 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:51 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:50 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 07:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 07:48 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog1002.eqiad.wmnet |
[production] |
| 07:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 07:48 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1117.eqiad.wmnet with OS bullseye |
[production] |
| 07:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 07:48 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mwlog1002.eqiad.wmnet |
[production] |
| 07:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1127 (T344589)', diff saved to https://phabricator.wikimedia.org/P50815 and previous config saved to /var/cache/conftool/dbconfig/20230822-074725-ladsgroup.json |
[production] |
| 07:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance |
[production] |