|
2022-11-07
ยง
|
| 08:36 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38189 and previous config saved to /var/cache/conftool/dbconfig/20221107-083626-marostegui.json |
[production] |
| 08:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:31 |
<urbanecm@deploy1002> |
urbanecm and kartik: Backport for [[gerrit:853914|ContentTranslation: Move haw, ps and xh Wikipedias out of Beta]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 08:31 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:853914|ContentTranslation: Move haw, ps and xh Wikipedias out of Beta]] |
[production] |
| 08:30 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:852924|Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)]] (duration: 06m 24s) |
[production] |
| 08:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:24 |
<urbanecm@deploy1002> |
urbanecm and kartik: Backport for [[gerrit:852924|Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 08:24 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:852924|Set ContentTranslation MT threshold to 75 in Japanese WP (T321819)]] |
[production] |
| 08:23 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:845058|Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)]] (duration: 18m 54s) |
[production] |
| 08:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38188 and previous config saved to /var/cache/conftool/dbconfig/20221107-082120-marostegui.json |
[production] |
| 08:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P38185 and previous config saved to /var/cache/conftool/dbconfig/20221107-080613-marostegui.json |
[production] |
| 08:05 |
<urbanecm@deploy1002> |
urbanecm and daniel: Backport for [[gerrit:845058|Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 08:04 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:845058|Set VisualEditorDefaultParsoidClient for dewiki-beta mad testwiki (T320531)]] |
[production] |
| 08:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 08:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 08:00 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:853910|ProductionServices.php: Promote pc2014 to pc3 master]] (duration: 04m 18s) |
[production] |
| 07:56 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:853910|ProductionServices.php: Promote pc2014 to pc3 master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 07:55 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:853910|ProductionServices.php: Promote pc2014 to pc3 master]] |
[production] |
| 07:51 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38184 and previous config saved to /var/cache/conftool/dbconfig/20221107-075106-marostegui.json |
[production] |
| 07:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:50 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T321123)', diff saved to https://phabricator.wikimedia.org/P38183 and previous config saved to /var/cache/conftool/dbconfig/20221107-074959-marostegui.json |
[production] |
| 07:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38182 and previous config saved to /var/cache/conftool/dbconfig/20221107-074938-marostegui.json |
[production] |
| 07:47 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] (duration: 04m 53s) |
[production] |
| 07:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:44 |
<urbanecm> |
Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (T318457) |
[production] |
| 07:42 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 07:42 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] |
[production] |
| 07:37 |
<elukey> |
`elukey@aux-k8s-worker1002:~$ sudo systemctl reset-failed ifup@ens13.service` |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38181 and previous config saved to /var/cache/conftool/dbconfig/20221107-073431-marostegui.json |
[production] |
| 07:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |