|
2022-11-17
ยง
|
| 13:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1188 (T323214)', diff saved to https://phabricator.wikimedia.org/P40109 and previous config saved to /var/cache/conftool/dbconfig/20221117-134753-ladsgroup.json |
[production] |
| 13:46 |
<moritzm> |
failover ganeti master in codfw to ganeti2021 |
[production] |
| 13:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet |
[production] |
| 13:32 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet |
[production] |
| 13:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet |
[production] |
| 13:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1188 (T323214)', diff saved to https://phabricator.wikimedia.org/P40108 and previous config saved to /var/cache/conftool/dbconfig/20221117-131709-ladsgroup.json |
[production] |
| 13:17 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T323214)', diff saved to https://phabricator.wikimedia.org/P40107 and previous config saved to /var/cache/conftool/dbconfig/20221117-131647-ladsgroup.json |
[production] |
| 13:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet |
[production] |
| 13:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet |
[production] |
| 13:01 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet |
[production] |
| 13:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P40106 and previous config saved to /var/cache/conftool/dbconfig/20221117-130141-ladsgroup.json |
[production] |
| 12:55 |
<mfossati@deploy1002> |
Finished deploy [airflow-dags/platform_eng@4bdda20]: (no justification provided) (duration: 00m 18s) |
[production] |
| 12:55 |
<mfossati@deploy1002> |
Started deploy [airflow-dags/platform_eng@4bdda20]: (no justification provided) |
[production] |
| 12:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P40105 and previous config saved to /var/cache/conftool/dbconfig/20221117-124634-ladsgroup.json |
[production] |
| 12:32 |
<mfossati@deploy1002> |
Finished deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided) (duration: 00m 05s) |
[production] |
| 12:32 |
<mfossati@deploy1002> |
Started deploy [airflow-dags/platform_eng@3bb99c2]: (no justification provided) |
[production] |
| 12:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T323214)', diff saved to https://phabricator.wikimedia.org/P40104 and previous config saved to /var/cache/conftool/dbconfig/20221117-123128-ladsgroup.json |
[production] |
| 12:30 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet |
[production] |
| 12:29 |
<moritzm> |
installing bluez security updates |
[production] |
| 12:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P40103 and previous config saved to /var/cache/conftool/dbconfig/20221117-122532-ladsgroup.json |
[production] |
| 12:24 |
<moritzm> |
restarting slapd on serpens/seaborgium/ldap-corp* to pick up GNUTLS update |
[production] |
| 12:23 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet |
[production] |
| 12:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad |
[production] |
| 12:18 |
<jmm@cumin2002> |
START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-eqiad |
[production] |
| 12:13 |
<sukhe> |
rolling restart of A:wikidough to pick up security updates |
[production] |
| 12:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet |
[production] |
| 12:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.maps.roll-restart (exit_code=0) rolling restart_daemons on A:maps-replica-codfw |
[production] |
| 12:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P40101 and previous config saved to /var/cache/conftool/dbconfig/20221117-121026-ladsgroup.json |
[production] |
| 12:06 |
<Emperor> |
restart swift proxies to deploy phonos changes to rewrite.py T317417 |
[production] |
| 12:06 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet |
[production] |
| 12:02 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=trwiki # T318457 |
[production] |
| 12:01 |
<hashar> |
Gerrit back since 11:45 UTC |
[production] |
| 12:01 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=enwiki # T318457 |
[production] |
| 11:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P40100 and previous config saved to /var/cache/conftool/dbconfig/20221117-115520-ladsgroup.json |
[production] |
| 11:50 |
<jmm@cumin2002> |
START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-replica-codfw |
[production] |
| 11:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet |
[production] |
| 11:47 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp5032.eqsin.wmnet |
[production] |
| 11:47 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp5032.eqsin.wmnet |
[production] |
| 11:41 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P40099 and previous config saved to /var/cache/conftool/dbconfig/20221117-114013-ladsgroup.json |
[production] |
| 11:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2175 (T323214)', diff saved to https://phabricator.wikimedia.org/P40098 and previous config saved to /var/cache/conftool/dbconfig/20221117-113814-ladsgroup.json |
[production] |
| 11:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1166 (T318955)', diff saved to https://phabricator.wikimedia.org/P40097 and previous config saved to /var/cache/conftool/dbconfig/20221117-113621-ladsgroup.json |
[production] |
| 11:36 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:36 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2024.codfw.wmnet |
[production] |
| 11:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2024.codfw.wmnet |
[production] |
| 11:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P40096 and previous config saved to /var/cache/conftool/dbconfig/20221117-112307-ladsgroup.json |
[production] |
| 11:20 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet |
[production] |