|
2023-08-22
ยง
|
| 09:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2178 (T343718)', diff saved to https://phabricator.wikimedia.org/P50850 and previous config saved to /var/cache/conftool/dbconfig/20230822-090026-ladsgroup.json |
[production] |
| 09:00 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50849 and previous config saved to /var/cache/conftool/dbconfig/20230822-090016-ladsgroup.json |
[production] |
| 08:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2120 (T344589)', diff saved to https://phabricator.wikimedia.org/P50848 and previous config saved to /var/cache/conftool/dbconfig/20230822-085332-ladsgroup.json |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1213:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50847 and previous config saved to /var/cache/conftool/dbconfig/20230822-084724-ladsgroup.json |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210 (T343718)', diff saved to https://phabricator.wikimedia.org/P50846 and previous config saved to /var/cache/conftool/dbconfig/20230822-084713-ladsgroup.json |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2103 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P50845 and previous config saved to /var/cache/conftool/dbconfig/20230822-084712-root.json |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2120 (T344589)', diff saved to https://phabricator.wikimedia.org/P50844 and previous config saved to /var/cache/conftool/dbconfig/20230822-084703-ladsgroup.json |
[production] |
| 08:46 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 08:46 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 08:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T344589)', diff saved to https://phabricator.wikimedia.org/P50843 and previous config saved to /var/cache/conftool/dbconfig/20230822-084638-ladsgroup.json |
[production] |
| 08:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2021', diff saved to https://phabricator.wikimedia.org/P50842 and previous config saved to /var/cache/conftool/dbconfig/20230822-084550-ladsgroup.json |
[production] |
| 08:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1136 (T344589)', diff saved to https://phabricator.wikimedia.org/P50841 and previous config saved to /var/cache/conftool/dbconfig/20230822-084536-ladsgroup.json |
[production] |
| 08:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P50840 and previous config saved to /var/cache/conftool/dbconfig/20230822-084510-ladsgroup.json |
[production] |
| 08:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1135 (T336886)', diff saved to https://phabricator.wikimedia.org/P50839 and previous config saved to /var/cache/conftool/dbconfig/20230822-084445-ladsgroup.json |
[production] |
| 08:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:43 |
<vgutierrez> |
restart ATS on cp5024 to clean the ATS restart alert - T344674 |
[production] |
| 08:42 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 08:42 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 08:42 |
<fabfur@cumin2002> |
START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo and A:cp |
[production] |
| 08:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P50838 and previous config saved to /var/cache/conftool/dbconfig/20230822-084104-ladsgroup.json |
[production] |
| 08:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1136 (T344589)', diff saved to https://phabricator.wikimedia.org/P50837 and previous config saved to /var/cache/conftool/dbconfig/20230822-083912-ladsgroup.json |
[production] |
| 08:39 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:38 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127 (T344589)', diff saved to https://phabricator.wikimedia.org/P50836 and previous config saved to /var/cache/conftool/dbconfig/20230822-083848-ladsgroup.json |
[production] |
| 08:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P50835 and previous config saved to /var/cache/conftool/dbconfig/20230822-083207-ladsgroup.json |
[production] |
| 08:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P50834 and previous config saved to /var/cache/conftool/dbconfig/20230822-083132-ladsgroup.json |
[production] |
| 08:31 |
<urbanecm> |
mwmaint1002: Stop frwiki instance of T315510 scripts due to a large volume of T343859 errors |
[production] |
| 08:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2021', diff saved to https://phabricator.wikimedia.org/P50833 and previous config saved to /var/cache/conftool/dbconfig/20230822-083044-ladsgroup.json |
[production] |
| 08:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315', diff saved to https://phabricator.wikimedia.org/P50832 and previous config saved to /var/cache/conftool/dbconfig/20230822-083004-ladsgroup.json |
[production] |
| 08:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P50831 and previous config saved to /var/cache/conftool/dbconfig/20230822-082559-ladsgroup.json |
[production] |
| 08:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P50830 and previous config saved to /var/cache/conftool/dbconfig/20230822-082342-ladsgroup.json |
[production] |
| 08:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P50829 and previous config saved to /var/cache/conftool/dbconfig/20230822-081701-ladsgroup.json |
[production] |
| 08:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P50828 and previous config saved to /var/cache/conftool/dbconfig/20230822-081626-ladsgroup.json |
[production] |
| 08:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50827 and previous config saved to /var/cache/conftool/dbconfig/20230822-081537-ladsgroup.json |
[production] |
| 08:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2171:3315 (T343718)', diff saved to https://phabricator.wikimedia.org/P50826 and previous config saved to /var/cache/conftool/dbconfig/20230822-081458-ladsgroup.json |
[production] |
| 08:12 |
<moritzm> |
bounce ferm on aux-k8s-ctrl1001 |
[production] |
| 08:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P50825 and previous config saved to /var/cache/conftool/dbconfig/20230822-081054-ladsgroup.json |
[production] |
| 08:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P50824 and previous config saved to /var/cache/conftool/dbconfig/20230822-080836-ladsgroup.json |
[production] |
| 08:05 |
<moritzm> |
installing Linux 4.19.289-2 on Buster hosts |
[production] |
| 08:05 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1117.eqiad.wmnet with reason: host reimage |
[production] |
| 08:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling es2021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50823 and previous config saved to /var/cache/conftool/dbconfig/20230822-080413-ladsgroup.json |
[production] |
| 08:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling es1021 (T344589)', diff saved to https://phabricator.wikimedia.org/P50822 and previous config saved to /var/cache/conftool/dbconfig/20230822-080328-ladsgroup.json |
[production] |
| 08:03 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1021.eqiad.wmnet with reason: Maintenance |
[production] |