|
2021-04-01
ยง
|
| 22:49 |
<twentyafterfour> |
deploying phatality |
[production] |
| 22:34 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2248.codfw.wmnet |
[production] |
| 22:33 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2247.codfw.wmnet |
[production] |
| 22:17 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2247.codfw.wmnet |
[production] |
| 22:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2246.codfw.wmnet |
[production] |
| 22:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2246.codfw.wmnet |
[production] |
| 21:58 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2243.codfw.wmnet |
[production] |
| 21:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2243.codfw.wmnet |
[production] |
| 20:42 |
<mutante> |
mw2243, mw2246, mw2247, mw2248 - depooled - replaced by mw2379, mw2380, mw2381, mw2382 ( T277780) |
[production] |
| 20:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2248.codfw.wmnet |
[production] |
| 20:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2247.codfw.wmnet |
[production] |
| 20:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2246.codfw.wmnet |
[production] |
| 20:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2243.codfw.wmnet |
[production] |
| 20:23 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2382.codfw.wmnet |
[production] |
| 20:23 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2381.codfw.wmnet |
[production] |
| 20:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2380.codfw.wmnet |
[production] |
| 20:20 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2379.codfw.wmnet |
[production] |
| 20:06 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2379.codfw.wmnet with reason: new_install |
[production] |
| 20:06 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2379.codfw.wmnet with reason: new_install |
[production] |
| 20:06 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2380.codfw.wmnet with reason: new_install |
[production] |
| 20:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2380.codfw.wmnet with reason: new_install |
[production] |
| 20:05 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2382.codfw.wmnet with reason: new_install |
[production] |
| 20:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2382.codfw.wmnet with reason: new_install |
[production] |
| 20:05 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2381.codfw.wmnet with reason: new_install |
[production] |
| 20:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2381.codfw.wmnet with reason: new_install |
[production] |
| 20:01 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 04s) |
[production] |
| 20:01 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 20:01 |
<razzi@deploy1002> |
deploy aborted: Deployment of superset fd7c9eb71e193, released after 1.0.1hv (duration: 00m 00s) |
[production] |
| 20:01 |
<mutante> |
mw2379, mw2380, mw2381, mw2382 - scap pull |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2382.codfw.wmnet |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2381.codfw.wmnet |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2380.codfw.wmnet |
[production] |
| 19:59 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 21s) |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2379.codfw.wmnet |
[production] |
| 19:58 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 19:57 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 19:57 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:56 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 12s) |
[production] |
| 19:56 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 19:51 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 19:37 |
<mutante> |
pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524) |
[production] |
| 19:34 |
<mutante> |
mw2379, mw2380, mw2381, mw2382 - rebooting |
[production] |
| 19:34 |
<twentyafterfour@deploy1002> |
scap sync-l10n completed (1.36.0-wmf.37) (duration: 02m 38s) |
[production] |
| 19:30 |
<mutante> |
depooled parse2001 because on train deployment it caused "MWException: No localisation cache found for English" and then "HTTP CRITICAL: HTTP/1.1 500 Internal Server Error" (T268524) |
[production] |
| 19:29 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:28 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:21 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.36.0-wmf.37 refs T278343 |
[production] |