|
2021-04-02
ยง
|
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2384.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2383.codfw.wmnet |
[production] |
| 21:19 |
<mutante> |
generating mcrouter certs for mw2395 through mw2404 (T278396) |
[production] |
| 21:07 |
<mutante> |
mw2383 through mw2394 - 'uptime && scap pull' via ssh -C (not cumin because it needs to run as non-root) |
[production] |
| 20:58 |
<mutante> |
mw238* - scap pull via cumin not possible because it doesnt work as root |
[production] |
| 20:50 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@86c7cdc]: tweak to affinity group options (duration: 03m 39s) |
[production] |
| 20:46 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@86c7cdc]: tweak to affinity group options |
[production] |
| 20:44 |
<mutante> |
mw2385 through mw2394 - serial rebooting |
[production] |
| 20:43 |
<mutante> |
mw2384 reboot |
[production] |
| 20:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 10 hosts with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 10 hosts with reason: new_install |
[production] |
| 20:40 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@86c7cdc]: update horizon for codfw1dev (duration: 01m 47s) |
[production] |
| 20:39 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@86c7cdc]: update horizon for codfw1dev |
[production] |
| 20:09 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 20:09 |
<bstorm@cumin1001> |
Added views for new wiki: taywiki T275836 |
[production] |
| 19:47 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 19:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2383.codfw.wmnet with reason: new_install |
[production] |
| 19:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2383.codfw.wmnet with reason: new_install |
[production] |
| 19:07 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 19:07 |
<bstorm@cumin1001> |
Added views for new wiki: mnwwiktionary T276126 |
[production] |
| 18:44 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 18:44 |
<mutante> |
[puppetmaster1001:~] $ sudo puppet node deactivate mw2247.codfw.wmnet |
[production] |
| 18:28 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2247.codfw.wmnet |
[production] |
| 18:20 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2247.codfw.wmnet |
[production] |
| 17:57 |
<legoktm> |
upgraded mailman3 python3-django-postorius on lists1002 |
[production] |
| 15:48 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 15:48 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 15:45 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 15:45 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 15:41 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 14:35 |
<jiji@cumin1001> |
conftool action : set/weight=20; selector: cluster=jobrunner,name=mw133[7-8].eqiad.wmnet |
[production] |
| 14:34 |
<jiji@cumin1001> |
conftool action : set/weight=20; selector: cluster=videoscaler,name=mw133[5-6].eqiad.wmnet |
[production] |
| 14:32 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw133[5-6].eqiad.wmnet |
[production] |
| 14:31 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw133[7-8].eqiad.wmnet |
[production] |
| 14:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:29 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw1111.eqiad.wmnet |
[production] |
| 14:28 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:20 |
<Urbanecm> |
Start server-side upload for 3 video files (T279060, T279061, T279062) |
[production] |
| 14:09 |
<Urbanecm> |
Start server-side upload for 3 video files (T279138, T279137, T279136) |
[production] |
| 13:42 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.37 |
[production] |
| 13:14 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 13:12 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 13:11 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/load.php: T278579 (duration: 00m 58s) |
[production] |
| 13:10 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/includes/OutputHandler.php: T278579 (duration: 00m 57s) |
[production] |
| 13:08 |
<reedy@deploy1002> |
Synchronized php-1.36.0-wmf.37/includes/MediaWiki.php: T278579 (duration: 00m 58s) |
[production] |
| 11:46 |
<Urbanecm> |
correction: Start server-side upload for 3 video files (T279079, T279080, T279104) |
[production] |
| 11:45 |
<Urbanecm> |
Start server-side upload for 3 images (T279079, T279080, T279104) |
[production] |
| 10:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1002.eqiad.wmnet with reason: REIMAGE |
[production] |