|
2021-04-02
§
|
| 22:31 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 22:31 |
<bstorm@cumin1001> |
Added views for new wiki: trvwiki T276246 |
[production] |
| 22:08 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 22:08 |
<mutante> |
pooled mw2395,mw2396 as API appservers running on new hardware |
[production] |
| 22:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw239[5-6].codfw.wmnet |
[production] |
| 21:58 |
<legoktm> |
legoktm@lists1002:~$ time sudo mailman-web rebuild_index |
[production] |
| 21:56 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw239[5-6].codfw.wmnet |
[production] |
| 21:55 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw239[5-6].codfw.wmnet |
[production] |
| 21:48 |
<mutante> |
mw2395, mw2396 - reboot - becoming API servers |
[production] |
| 21:43 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw239[0-4].codfw.wmnet |
[production] |
| 21:42 |
<mutante> |
pooled 12 brand-new codfw appservers running on new hardware generation |
[production] |
| 21:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw238[5-9].codfw.wmnet |
[production] |
| 21:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2384.codfw.wmnet |
[production] |
| 21:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2383.codfw.wmnet |
[production] |
| 21:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install |
[production] |
| 21:38 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install |
[production] |
| 21:37 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:35 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:35 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:35 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw239[0-4].codfw.wmnet |
[production] |
| 21:34 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw238[3-9].codfw.wmnet |
[production] |
| 21:33 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:28 |
<legoktm> |
imported python-xapian-haystack 2.1.0-6~wmf1 on apt1001 (T278717) |
[production] |
| 21:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2394.codfw.wmnet |
[production] |
| 21:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2393.codfw.wmnet |
[production] |
| 21:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2392.codfw.wmnet |
[production] |
| 21:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2391.codfw.wmnet |
[production] |
| 21:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2390.codfw.wmnet |
[production] |
| 21:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2389.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2388.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2387.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2386.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2385.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2384.codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2383.codfw.wmnet |
[production] |
| 21:19 |
<mutante> |
generating mcrouter certs for mw2395 through mw2404 (T278396) |
[production] |
| 21:07 |
<mutante> |
mw2383 through mw2394 - 'uptime && scap pull' via ssh -C (not cumin because it needs to run as non-root) |
[production] |
| 20:58 |
<mutante> |
mw238* - scap pull via cumin not possible because it doesnt work as root |
[production] |
| 20:50 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@86c7cdc]: tweak to affinity group options (duration: 03m 39s) |
[production] |
| 20:46 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@86c7cdc]: tweak to affinity group options |
[production] |
| 20:44 |
<mutante> |
mw2385 through mw2394 - serial rebooting |
[production] |
| 20:43 |
<mutante> |
mw2384 reboot |
[production] |
| 20:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 10 hosts with reason: new_install |
[production] |
| 20:43 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 10 hosts with reason: new_install |
[production] |
| 20:40 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@86c7cdc]: update horizon for codfw1dev (duration: 01m 47s) |
[production] |