|
2020-11-18
ยง
|
| 19:19 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:13 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: gerrit:641527 - Set to 0 (duration: 01m 04s) |
[production] |
| 18:26 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:26 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:44 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:38 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:18 |
<elukey> |
shutdown an-presto1004 for hw maintenance |
[production] |
| 17:13 |
<akosiaris> |
T241230 pool codfw kubernetes for recommendation-api at a very low weight |
[production] |
| 17:12 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 17:12 |
<akosiaris@cumin1001> |
conftool action : set/weight=1; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 16:52 |
<jbond42> |
drop os_version/requiers_os functions from wmflib |
[production] |
| 16:50 |
<elukey> |
update /etc/krb5.keytab on krb1001/krb2001 to match the most up to date key version for host/krb2001.codfw.wmnet |
[production] |
| 16:49 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:49 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:44 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:43 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:38 |
<reedy@deploy1001> |
Synchronized wmf-config/logging.php: T268141 (duration: 01m 06s) |
[production] |
| 16:36 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:32 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:31 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:27 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:59 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:56 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:51 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:42 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:16 |
<Urbanecm> |
mwscript deleteEqualMessages.php --wiki=cswiki --delete |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:12 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:12 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 15:12 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 15:11 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 15:09 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 15:09 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 15:05 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 15:03 |
<Urbanecm> |
Purge https://2030.wikimedia.org/ via purgeList.php (T264797) |
[production] |
| 14:34 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:30 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:30 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:17 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:13 |
<Urbanecm> |
Purge https://2030.wikimedia.org/ via purgeList.php (T264797) |
[production] |
| 14:09 |
<elukey> |
copied /etc/krb5.keytab from krb1001 to krb2001 (the last one contained only one principal for 2001, the first one both for 1001 and 2001) |
[production] |
| 14:05 |
<moritzm> |
installing openldap security updates on ro replicas |
[production] |
| 14:03 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:02 |
<elukey> |
restart krb5-kpropd.service on krb2001 to force the pick up of new client configs |
[production] |
| 13:35 |
<bblack> |
cache_text: Executing "varnishadm -n frontend param.set nuke_limit 1000" - T266373 |
[production] |
| 13:34 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 13:30 |
<moritzm> |
installing openldap security updates on corp replicas |
[production] |
| 13:08 |
<Urbanecm> |
EU B&C done (~15 minutes ago) |
[production] |
| 12:43 |
<akosiaris> |
sync staging cluster's helmfile.d/admin state. Aside from calico, the rest is a noop |
[production] |
| 12:43 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |