|
2022-10-11
§
|
| 07:31 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 07:30 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 07:24 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 07:22 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 07:21 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . |
[production] |
| 07:21 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . |
[production] |
| 07:18 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 07:18 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 07:17 |
<ryankemper> |
[Elastic] Forcing recheck of elastic settings check alerts; expecting a bit of noise as the alerts resolve (hopefully) |
[production] |
| 07:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet |
[production] |
| 07:17 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 07:16 |
<ryankemper> |
[Elastic] Updated cross-cluster remote seeds (masters): `ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.eqiad.wmnet:9[2,4,6]43/_cluster/settings --ccc chi=chi_eqiad_masters.lst psi=psi_eqiad_masters.lst omega=omega_eqiad_masters.lst` |
[production] |
| 07:15 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 07:12 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 07:11 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 07:09 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:839411|ContentTranslation: Make Mongolian Wikipedia MT stricter by 10% (T319156)]] (duration: 08m 56s) |
[production] |
| 07:01 |
<kartik@deploy1002> |
kartik and kartik: Backport for [[gerrit:839411|ContentTranslation: Make Mongolian Wikipedia MT stricter by 10% (T319156)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 07:01 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:839411|ContentTranslation: Make Mongolian Wikipedia MT stricter by 10% (T319156)]] |
[production] |
| 06:54 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 06:53 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 06:53 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 06:52 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 06:44 |
<elukey> |
kill leftover process of jmads on stat1005 to allow user cleanup via puppet |
[production] |
| 06:43 |
<elukey> |
kill leftover process of nokafor on stat1004 to allow user cleanup via puppet |
[production] |
| 06:37 |
<elukey> |
kill leftover process of bmansurov on stat1007 to allow user cleanup via puppet |
[production] |
| 06:35 |
<XioNoX> |
delete now unused VC ports on asw2-c4-eqiad - T313384 |
[production] |
| 06:34 |
<elukey> |
kill leftover process of bmansurov on an-airflow1002 to allow user cleanup via puppet |
[production] |
| 03:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 03:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 03:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 03:06 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
|
2022-10-10
§
|
| 21:19 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS bullseye |
[production] |
| 20:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:44 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:841151|Resize wordmark and tagline of Bengali Wikibooks (T319320)]] (duration: 07m 29s) |
[production] |
| 19:16 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:14 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:14 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns4004 |
[production] |
| 19:14 |
<robh@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dns4004.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 19:13 |
<robh@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host dns4004 |
[production] |
| 19:07 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host dns4004.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 18:39 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns4002.wikimedia.org |
[production] |
| 18:39 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:38 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4008.ulsfo.wmnet with OS bullseye |
[production] |
| 18:35 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:30 |
<robh@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts dns4002.wikimedia.org |
[production] |
| 18:21 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage |
[production] |