1-50 of 10000 results (26ms)
2021-02-12 ยง
23:59 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1348.eqiad.wmnet [production]
23:58 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1356.eqiad.wmnet [production]
23:43 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1284.eqiad.wmnet [production]
23:42 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1283.eqiad.wmnet [production]
23:42 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1282.eqiad.wmnet [production]
23:42 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1348.eqiad.wmnet [production]
23:41 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1356.eqiad.wmnet [production]
23:41 <legoktm@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1221.eqiad.wmnet [production]
23:39 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1221.eqiad.wmnet [production]
23:38 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1281.eqiad.wmnet [production]
23:26 <dduvall@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:24 <dduvall@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:14 <dduvall@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
23:02 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
22:52 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
22:51 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1284.eqiad.wmnet with reason: REIMAGE [production]
22:49 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1283.eqiad.wmnet with reason: REIMAGE [production]
22:48 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1284.eqiad.wmnet with reason: REIMAGE [production]
22:47 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1283.eqiad.wmnet with reason: REIMAGE [production]
22:47 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1282.eqiad.wmnet with reason: REIMAGE [production]
22:45 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1282.eqiad.wmnet with reason: REIMAGE [production]
22:44 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1281.eqiad.wmnet with reason: REIMAGE [production]
22:42 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1281.eqiad.wmnet with reason: REIMAGE [production]
22:32 <krinkle@deploy1001> Synchronized wmf-config/PoolCounterSettings.php: Idc385de0 cleanup (duration: 05m 14s) [production]
22:15 <krinkle@deploy1001> Synchronized wmf-config/etcd.php: b3447343a cleanup (duration: 05m 20s) [production]
22:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1348.eqiad.wmnet with reason: REIMAGE [production]
21:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1348.eqiad.wmnet with reason: REIMAGE [production]
21:26 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1357.eqiad.wmnet [production]
20:52 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1357.eqiad.wmnet with reason: REIMAGE [production]
20:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1356.eqiad.wmnet with reason: REIMAGE [production]
20:48 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1357.eqiad.wmnet with reason: REIMAGE [production]
20:48 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1356.eqiad.wmnet with reason: REIMAGE [production]
20:36 <mutante> mwdebug1003 now on buster - mwdebug1002 rebooting and reimaging to buster [production]
20:36 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mwdebug1002.eqiad.wmnet with reason: OS upgrade [production]
20:35 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mwdebug1002.eqiad.wmnet with reason: OS upgrade [production]
20:32 <mutante> mw1353, mw1358 - scap pull, repooled [production]
20:30 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1353.eqiad.wmnet [production]
20:30 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1358.eqiad.wmnet [production]
20:17 <mutante> mwdebug2001 - restarted memcached [production]
20:09 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1358.eqiad.wmnet [production]
20:09 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1353.eqiad.wmnet [production]
19:56 <mutante> mwdebug2002 - restart memcached [production]
19:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw1358.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw1358.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw1353.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw1353.eqiad.wmnet with reason: OS upgrade [production]
19:46 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mwdebug1003.eqiad.wmnet with reason: OS upgrade [production]
19:46 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mwdebug1003.eqiad.wmnet with reason: OS upgrade [production]
19:43 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: roll back commonswiki to 1.36.0-wmf.27 due to T274589 [production]
19:42 <mutante> mwdebug2001 now on buster - mwdebug1003 rebooting and reimaging to stretch [production]