|
2021-03-30
ยง
|
| 20:58 |
<legoktm> |
killed remaining ffmpeg on mw1298 |
[production] |
| 20:56 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 12s) |
[production] |
| 20:56 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:53 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 20:52 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 20:41 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 20s) |
[production] |
| 20:41 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:41 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 20:40 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 20:38 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 20:37 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 20:37 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 31s) |
[production] |
| 20:36 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:35 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 05s) |
[production] |
| 20:35 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:34 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 20:34 |
<twentyafterfour@deploy1002> |
Started restart [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:33 |
<twentyafterfour@deploy1002> |
Finished scap: testwikis wikis to 1.36.0-wmf.37 refs T278343 (duration: 80m 32s) |
[production] |
| 20:29 |
<twentyafterfour@deploy1002> |
Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 49s) |
[production] |
| 20:29 |
<twentyafterfour@deploy1002> |
Started deploy [releng/phatality@715d809]: (no justification provided) |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1307.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1306.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1305.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1304.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1303.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1307.eqiad.wmnet |
[production] |
| 20:28 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1306.eqiad.wmnet |
[production] |
| 20:27 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1305.eqiad.wmnet |
[production] |
| 20:27 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1304.eqiad.wmnet |
[production] |
| 20:27 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1303.eqiad.wmnet |
[production] |
| 20:26 |
<twentyafterfour> |
preparing to deploy phatality upgrade to kibana cluster |
[production] |
| 20:25 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1296.eqiad.wmnet |
[production] |
| 20:25 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1298.eqiad.wmnet |
[production] |
| 20:25 |
<legoktm@deploy1002> |
conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1299.eqiad.wmnet |
[production] |
| 20:21 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@1a53e9a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1a53e9a] (duration: 04m 29s) |
[production] |
| 20:20 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw1299.eqiad.wmnet |
[production] |
| 20:20 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw1298.eqiad.wmnet |
[production] |
| 20:20 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw1296.eqiad.wmnet |
[production] |
| 20:16 |
<joal@deploy1002> |
Started deploy [analytics/refinery@1a53e9a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1a53e9a] |
[production] |
| 20:16 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@1a53e9a] (thin): Regular analytics weekly train THIN [analytics/refinery@1a53e9a] (duration: 00m 07s) |
[production] |
| 20:16 |
<joal@deploy1002> |
Started deploy [analytics/refinery@1a53e9a] (thin): Regular analytics weekly train THIN [analytics/refinery@1a53e9a] |
[production] |
| 20:15 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@1a53e9a]: Regular analytics weekly train [analytics/refinery@1a53e9a] (duration: 17m 11s) |
[production] |
| 20:02 |
<twentyafterfour> |
when syncing 1.36.0-wmf.37 promote to testwikis, one server failed: server mw1298.eqiad.wmnet and two more appear to be hung because scap is stuck at 2 left 99% without making any progress for a long time now. refs T278343 |
[production] |
| 19:58 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet |
[production] |
| 19:58 |
<joal@deploy1002> |
Started deploy [analytics/refinery@1a53e9a]: Regular analytics weekly train [analytics/refinery@1a53e9a] |
[production] |
| 19:58 |
<bblack> |
repool cp1087 - T278729 |
[production] |
| 19:13 |
<twentyafterfour@deploy1002> |
Started scap: testwikis wikis to 1.36.0-wmf.37 refs T278343 |
[production] |
| 18:15 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:09 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:22 |
<legoktm> |
moved mw[1293-1295] to jobrunners and mw[1300-1302] to videoscalers |
[production] |