6751-6800 of 10000 results (40ms)
2021-04-05 §
06:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1181 in s7 with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15113 and previous config saved to /var/cache/conftool/dbconfig/20210405-064727-marostegui.json [production]
05:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1181 in s7 with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15112 and previous config saved to /var/cache/conftool/dbconfig/20210405-054951-marostegui.json [production]
05:30 <marostegui> Deploy schema change on db1121, lag will appear on s4 on wikireplicas [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 for schema change', diff saved to https://phabricator.wikimedia.org/P15111 and previous config saved to /var/cache/conftool/dbconfig/20210405-053000-marostegui.json [production]
05:12 <marostegui> Restart all sanitarium hosts to pick up new filters T278573 [production]
2021-04-04 §
14:47 <andrew@deploy1002> Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s) [production]
14:45 <andrew@deploy1002> Started deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch [production]
2021-04-03 §
19:20 <andrew@deploy1002> Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s) [production]
19:18 <andrew@deploy1002> Started deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch [production]
17:30 <andrew@deploy1002> Finished deploy [horizon/deploy@3a84c77]: upgrade labtesthorizon to the Wallaby branch (duration: 03m 35s) [production]
17:26 <andrew@deploy1002> Started deploy [horizon/deploy@3a84c77]: upgrade labtesthorizon to the Wallaby branch [production]
16:44 <elukey> power reset for ms-be2028 - not reachable via ssh, no tty available via mgmt console, NMI unrecoverable errors logged in iLo's system logs [production]
15:35 <andrew@deploy1002> Finished deploy [horizon/deploy@3a84c77]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 18s) [production]
15:33 <andrew@deploy1002> Started deploy [horizon/deploy@3a84c77]: upgrade labtesthorizon to the Wallaby branch [production]
15:12 <andrew@deploy1002> Finished deploy [horizon/deploy@8833f80]: upgrade labtesthorizon to the Wallaby branch (duration: 11m 51s) [production]
15:00 <andrew@deploy1002> Started deploy [horizon/deploy@8833f80]: upgrade labtesthorizon to the Wallaby branch [production]
05:38 <andrew@deploy1002> Finished deploy [horizon/deploy@35199a3]: upgrade labtesthorizon to the Wallaby branch (duration: 03m 05s) [production]
05:35 <andrew@deploy1002> Started deploy [horizon/deploy@35199a3]: upgrade labtesthorizon to the Wallaby branch [production]
2021-04-02 §
22:31 <bstorm@cumin1001> END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) [production]
22:31 <bstorm@cumin1001> Added views for new wiki: trvwiki T276246 [production]
22:08 <bstorm@cumin1001> START - Cookbook wmcs.wikireplicas.add_wiki [production]
22:08 <mutante> pooled mw2395,mw2396 as API appservers running on new hardware [production]
22:08 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw239[5-6].codfw.wmnet [production]
21:58 <legoktm> legoktm@lists1002:~$ time sudo mailman-web rebuild_index [production]
21:56 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw239[5-6].codfw.wmnet [production]
21:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw239[5-6].codfw.wmnet [production]
21:48 <mutante> mw2395, mw2396 - reboot - becoming API servers [production]
21:43 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw239[0-4].codfw.wmnet [production]
21:42 <mutante> pooled 12 brand-new codfw appservers running on new hardware generation [production]
21:41 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw238[5-9].codfw.wmnet [production]
21:40 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2384.codfw.wmnet [production]
21:40 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2383.codfw.wmnet [production]
21:38 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install [production]
21:38 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[2395-2396].codfw.wmnet with reason: new_install [production]
21:37 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE [production]
21:35 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE [production]
21:35 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: REIMAGE [production]
21:35 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw239[0-4].codfw.wmnet [production]
21:34 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw238[3-9].codfw.wmnet [production]
21:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: REIMAGE [production]
21:28 <legoktm> imported python-xapian-haystack 2.1.0-6~wmf1 on apt1001 (T278717) [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2394.codfw.wmnet [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2393.codfw.wmnet [production]
21:24 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2392.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2391.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2390.codfw.wmnet [production]
21:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2389.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2388.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2387.codfw.wmnet [production]
21:21 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2386.codfw.wmnet [production]