6051-6100 of 10000 results (28ms)
2021-01-29 §
01:06 <mutante> repooled mw2048,mw2049 - jobrunners/videoscalers now on buster [production]
01:06 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2253.codfw.wmnet [production]
01:06 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2249.codfw.wmnet [production]
01:05 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2248.codfw.wmnet [production]
01:03 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2249.codfw.wmnet [production]
01:03 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2248.codfw.wmnet [production]
00:19 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2261.codfw.wmnet [production]
00:14 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2262.codfw.wmnet [production]
00:13 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2283.codfw.wmnet [production]
2021-01-28 §
23:58 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2261.codfw.wmnet [production]
23:58 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2262.codfw.wmnet [production]
23:57 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2283.codfw.wmnet [production]
23:52 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2253.codfw.wmnet with reason: REIMAGE [production]
23:49 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2253.codfw.wmnet with reason: REIMAGE [production]
23:47 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2248.codfw.wmnet with reason: REIMAGE [production]
23:45 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2249.codfw.wmnet with reason: REIMAGE [production]
23:44 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2248.codfw.wmnet with reason: REIMAGE [production]
23:43 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2249.codfw.wmnet with reason: REIMAGE [production]
23:34 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw2283.codfw.wmnet with reason: reimaging [production]
23:33 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mw2283.codfw.wmnet with reason: reimaging [production]
23:33 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2262.codfw.wmnet with reason: REIMAGE [production]
23:31 <legoktm@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2283.codfw.wmnet with reason: REIMAGE [production]
23:31 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2261.codfw.wmnet with reason: REIMAGE [production]
23:29 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2283.codfw.wmnet with reason: REIMAGE [production]
23:29 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2262.codfw.wmnet with reason: REIMAGE [production]
23:29 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2261.codfw.wmnet with reason: REIMAGE [production]
23:14 <mutante> reimaging jobrunners/videoscallers mw2248,mw2249 [production]
22:43 <brennen@deploy1001> Synchronized php-1.36.0-wmf.27/includes/parser/CacheTime.php: [[gerrit:658688|CacheTime: Extra protection for rollback unserialization (T273007)]] (duration: 00m 57s) [production]
22:41 <bblack> eqiad lvs should be back to normal state now with everything working [production]
22:39 <bblack> lvs1014 - apply https://gerrit.wikimedia.org/r/659439 [production]
22:37 <bblack> lvs1013 - testing https://gerrit.wikimedia.org/r/659439 (expect nop, worked on 1015!) [production]
22:36 <bblack> lvs1015 - testing https://gerrit.wikimedia.org/r/659439 (expect nop) [production]
22:21 <bblack> lvs1016 - trying https://gerrit.wikimedia.org/r/659439 on backup LVS... [production]
22:21 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2287.codfw.wmnet [production]
22:21 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2286.codfw.wmnet [production]
22:20 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2285.codfw.wmnet [production]
22:20 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw2284.codfw.wmnet [production]
22:16 <bblack> disabling puppet on all eqiad lvs for https://gerrit.wikimedia.org/r/659439 risks [production]
22:03 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2284.codfw.wmnet [production]
22:03 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2285.codfw.wmnet [production]
22:02 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2286.codfw.wmnet [production]
22:02 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw2287.codfw.wmnet [production]
21:33 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1173.eqiad.wmnet with reason: REIMAGE [production]
21:32 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: REIMAGE [production]
21:30 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1175.eqiad.wmnet with reason: REIMAGE [production]
21:28 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: REIMAGE [production]
21:28 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: REIMAGE [production]
21:28 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1175.eqiad.wmnet with reason: REIMAGE [production]
21:28 <brennen@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.28 [production]
21:28 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw2287.codfw.wmnet with reason: reimaging [production]