551-600 of 10000 results (37ms)
2021-04-02 ยง
21:48 <mutante> mw2395, mw2396 - reboot - becoming API servers [production]
21:43 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw239[0-4].codfw.wmnet [production]
21:42 <mutante> pooled 12 brand-new codfw appservers running on new hardware generation [production]
21:41 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw238[5-9].codfw.wmnet [production]
21:40 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2384.codfw.wmnet [production]
21:40 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2383.codfw.wmnet [production]
21:38 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install [production]
21:38 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install [production]
21:37 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE [production]
21:35 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE [production]
21:35 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE [production]
21:35 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw239[0-4].codfw.wmnet [production]
21:34 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw238[3-9].codfw.wmnet [production]
21:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE [production]
21:28 <legoktm> imported python-xapian-haystack 2.1.0-6~wmf1 on apt1001 (T278717) [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2394.codfw.wmnet [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2393.codfw.wmnet [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2392.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2391.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2390.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2389.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2388.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2387.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2386.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2385.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2384.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2383.codfw.wmnet [production]
21:19 <mutante> generating mcrouter certs for mw2395 through mw2404 (T278396) [production]
21:07 <mutante> mw2383 through mw2394 - 'uptime && scap pull' via ssh -C (not cumin because it needs to run as non-root) [production]
20:58 <mutante> mw238* - scap pull via cumin not possible because it doesnt work as root [production]
20:50 <andrew@deploy1002> Finished deploy [horizon/deploy@86c7cdc]: tweak to affinity group options (duration: 03m 39s) [production]
20:46 <andrew@deploy1002> Started deploy [horizon/deploy@86c7cdc]: tweak to affinity group options [production]
20:44 <mutante> mw2385 through mw2394 - serial rebooting [production]
20:43 <mutante> mw2384 reboot [production]
20:43 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install [production]
20:43 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2390-2394].codfw.wmnet with reason: new_install [production]
20:43 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 10 hosts with reason: new_install [production]
20:43 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 10 hosts with reason: new_install [production]
20:40 <andrew@deploy1002> Finished deploy [horizon/deploy@86c7cdc]: update horizon for codfw1dev (duration: 01m 47s) [production]
20:39 <andrew@deploy1002> Started deploy [horizon/deploy@86c7cdc]: update horizon for codfw1dev [production]
20:09 <bstorm@cumin1001> END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) [production]
20:09 <bstorm@cumin1001> Added views for new wiki: taywiki T275836 [production]
19:47 <bstorm@cumin1001> START - Cookbook wmcs.wikireplicas.add_wiki [production]
19:29 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2383.codfw.wmnet with reason: new_install [production]
19:29 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2383.codfw.wmnet with reason: new_install [production]
19:07 <bstorm@cumin1001> END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) [production]
19:07 <bstorm@cumin1001> Added views for new wiki: mnwwiktionary T276126 [production]
18:44 <bstorm@cumin1001> START - Cookbook wmcs.wikireplicas.add_wiki [production]
18:44 <mutante> [puppetmaster1001:~] $ sudo puppet node deactivate mw2247.codfw.wmnet [production]
18:28 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2247.codfw.wmnet [production]