|
2022-04-18
ยง
|
| 20:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25174 and previous config saved to /var/cache/conftool/dbconfig/20220418-205021-ladsgroup.json |
[production] |
| 20:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25173 and previous config saved to /var/cache/conftool/dbconfig/20220418-204619-ladsgroup.json |
[production] |
| 20:44 |
<AntiComposite> |
/cs flags #cvn-sw Vermont voiced |
[cvn] |
| 20:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P25172 and previous config saved to /var/cache/conftool/dbconfig/20220418-203755-ladsgroup.json |
[production] |
| 20:37 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:37 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25171 and previous config saved to /var/cache/conftool/dbconfig/20220418-203516-ladsgroup.json |
[production] |
| 20:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25170 and previous config saved to /var/cache/conftool/dbconfig/20220418-203114-ladsgroup.json |
[production] |
| 20:29 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:29 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25169 and previous config saved to /var/cache/conftool/dbconfig/20220418-202855-ladsgroup.json |
[production] |
| 20:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25168 and previous config saved to /var/cache/conftool/dbconfig/20220418-202011-ladsgroup.json |
[production] |
| 20:16 |
<wm-bot> |
<lokal-profil> Restarting webservice with the '--release buster' flag. php version compatibility needs to be investigated before migrating to kubernetes |
[tools.slumpartikel] |
| 20:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25167 and previous config saved to /var/cache/conftool/dbconfig/20220418-201609-ladsgroup.json |
[production] |
| 20:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25166 and previous config saved to /var/cache/conftool/dbconfig/20220418-201350-ladsgroup.json |
[production] |
| 20:10 |
<urbanecm> |
UTC late backport window done |
[production] |
| 20:09 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 0efb2b2: Add WikiEditor Realtime Preview to BetaFeatures (T304596) (duration: 00m 51s) |
[production] |
| 20:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25165 and previous config saved to /var/cache/conftool/dbconfig/20220418-200506-ladsgroup.json |
[production] |
| 20:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25164 and previous config saved to /var/cache/conftool/dbconfig/20220418-200418-ladsgroup.json |
[production] |
| 20:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:04 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25163 and previous config saved to /var/cache/conftool/dbconfig/20220418-200404-ladsgroup.json |
[production] |
| 20:02 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1021.eqiad.wmnet with OS buster |
[production] |
| 19:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25162 and previous config saved to /var/cache/conftool/dbconfig/20220418-195845-ladsgroup.json |
[production] |
| 19:56 |
<wm-bot> |
<lokal-profil> Adding the -release buster flag to cronjobs as part of Stretch deprecation |
[tools.heritage] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25161 and previous config saved to /var/cache/conftool/dbconfig/20220418-194859-ladsgroup.json |
[production] |
| 19:46 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage |
[production] |
| 19:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P25160 and previous config saved to /var/cache/conftool/dbconfig/20220418-194340-ladsgroup.json |
[production] |
| 19:43 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25159 and previous config saved to /var/cache/conftool/dbconfig/20220418-193354-ladsgroup.json |
[production] |
| 19:32 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.reimage for host clouddb1021.eqiad.wmnet with OS buster |
[production] |
| 19:27 |
<brennen> |
gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant |
[releng] |
| 19:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25158 and previous config saved to /var/cache/conftool/dbconfig/20220418-191849-ladsgroup.json |
[production] |
| 19:08 |
<mutante> |
- gitlab-prod-1001 is indeed back after soft rebooting the instance. uptime 1 min T297411 |
[devtools] |
| 19:07 |
<mutante> |
- gitlab-prod-1001 randomly stopped working. we got the "puppet failed" mails without having made changes and can't ssh to the instance anymore when trying to check out why. trying soft reboot via Horizon T297411 |
[devtools] |
| 19:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P25157 and previous config saved to /var/cache/conftool/dbconfig/20220418-190640-ladsgroup.json |
[production] |
| 19:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P25156 and previous config saved to /var/cache/conftool/dbconfig/20220418-190632-ladsgroup.json |
[production] |
| 19:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1179 (T298565)', diff saved to https://phabricator.wikimedia.org/P25155 and previous config saved to /var/cache/conftool/dbconfig/20220418-190452-ladsgroup.json |
[production] |
| 19:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:04 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298565)', diff saved to https://phabricator.wikimedia.org/P25154 and previous config saved to /var/cache/conftool/dbconfig/20220418-190444-ladsgroup.json |
[production] |
| 18:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25153 and previous config saved to /var/cache/conftool/dbconfig/20220418-185126-ladsgroup.json |
[production] |
| 18:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25152 and previous config saved to /var/cache/conftool/dbconfig/20220418-184939-ladsgroup.json |
[production] |