|
2020-11-18
ยง
|
| 23:34 |
<mutante> |
disabling puppet on memcache::mediawiki - deploying gerrit:637742 |
[production] |
| 22:56 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@6bbac6d]: Fix for bytes/str issue after T267269 (duration: 00m 04s) |
[production] |
| 22:56 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@6bbac6d]: Fix for bytes/str issue after T267269 |
[production] |
| 22:24 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:22 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:19 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Deploy GlobalWatchlist to beta (noop; T268181) (duration: 01m 04s) |
[production] |
| 22:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Deploy GlobalWatchlist extension: Prepare IS.php to know relevant variables (noop; T268181) (duration: 01m 06s) |
[production] |
| 22:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/extension-list: Deploy GlobalWatchlist extension to beta: add it to extension-list (T268181) (duration: 01m 05s) |
[production] |
| 21:53 |
<mutante> |
mwdebug1003 - restarting ferm because config was generated but service not restarted due to puppet dependency errors, breaking NRPE monitoring T267248 |
[production] |
| 21:47 |
<mutante> |
mwdebug1003 - scap pull - T267248 |
[production] |
| 21:40 |
<mutante> |
mw1317,mw1318 - back in action and all monitoring activated again |
[production] |
| 21:17 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1318.eqiad.wmnet,cluster=videoscaler |
[production] |
| 21:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet |
[production] |
| 21:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet |
[production] |
| 21:02 |
<mutante> |
mw1317,mw1318 - repooled=no after physical move to rack B |
[production] |
| 20:56 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1318.eqiad.wmnet |
[production] |
| 20:54 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1317.eqiad.wmnet |
[production] |
| 20:27 |
<mutante> |
mw1317, mw1318 shutting down for physical move |
[production] |
| 20:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw1318.eqiad.wmnet |
[production] |
| 20:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw1317.eqiad.wmnet |
[production] |
| 20:15 |
<mutante> |
mw1317,mw1318 - downtimed and depooled - they are physically moving from B7 to B5 (T266164) |
[production] |
| 20:13 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:13 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:13 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:13 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:11 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1317.eqiad.wmnet |
[production] |
| 20:11 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1318.eqiad.wmnet |
[production] |
| 20:10 |
<dancy@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.18 (duration: 01m 03s) |
[production] |
| 20:09 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.18 |
[production] |
| 20:03 |
<akosiaris@cumin1001> |
conftool action : set/pooled=inactive; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 20:03 |
<akosiaris@cumin1001> |
conftool action : set/weight=0; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 19:53 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 19:48 |
<otto@deploy1001> |
Synchronized php-1.36.0-wmf.16/extensions/EventLogging/modules/ext.eventLogging/core.js: EventLogging legacy events should use dt as server side receive time - T240460 (duration: 01m 06s) |
[production] |
| 19:45 |
<otto@deploy1001> |
Synchronized php-1.36.0-wmf.18/extensions/EventLogging/modules/ext.eventLogging/core.js: EventLogging legacy events should use dt as server side receive time - T240460 (duration: 01m 07s) |
[production] |
| 19:26 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:23 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:635607 - Switch ParserCache to JSON for group0 wikis (duration: 01m 05s) |
[production] |
| 19:19 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: gerrit:635086 - Enable parsoid on api_appserver (duration: 01m 04s) |
[production] |
| 19:19 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:13 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: gerrit:641527 - Set to 0 (duration: 01m 04s) |
[production] |
| 18:26 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:26 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:44 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:38 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:18 |
<elukey> |
shutdown an-presto1004 for hw maintenance |
[production] |
| 17:13 |
<akosiaris> |
T241230 pool codfw kubernetes for recommendation-api at a very low weight |
[production] |
| 17:12 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 17:12 |
<akosiaris@cumin1001> |
conftool action : set/weight=1; selector: service=recommendation-api,name=kubernetes.*,dc=codfw |
[production] |
| 16:52 |
<jbond42> |
drop os_version/requiers_os functions from wmflib |
[production] |
| 16:50 |
<elukey> |
update /etc/krb5.keytab on krb1001/krb2001 to match the most up to date key version for host/krb2001.codfw.wmnet |
[production] |
| 16:49 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |