3601-3650 of 10000 results (46ms)
2021-08-13 ยง
14:41 <mutante> mw1419 - started ferm [production]
13:35 <sukhe> ran homer for Gerrit 712400: Set up BGP peering to doh4002 in ulsfo [production]
13:23 <mutante> mw1453 - manual powercycle after it never rebooted when the reimage cookbook tries to trigger one [production]
13:22 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1450.eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 [production]
13:21 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1450.eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 [production]
13:21 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[1447-1449].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 [production]
13:21 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[1447-1449].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 [production]
12:54 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1454.eqiad.wmnet with reason: REIMAGE [production]
12:53 <godog> set runtime envoy.reloadable_features.strict_1xx_and_204_response_headers=false on thanos-fe* - T288815 [production]
12:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2001.codfw.wmnet with reason: new setup [production]
12:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2001.codfw.wmnet with reason: new setup [production]
12:52 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1454.eqiad.wmnet with reason: REIMAGE [production]
12:33 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1452.eqiad.wmnet with reason: REIMAGE [production]
12:31 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1451.eqiad.wmnet with reason: REIMAGE [production]
12:30 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1452.eqiad.wmnet with reason: REIMAGE [production]
12:29 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw1450.eqiad.wmnet with reason: new setup [production]
12:29 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mw1450.eqiad.wmnet with reason: new setup [production]
12:29 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1450.eqiad.wmnet with reason: REIMAGE [production]
12:28 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1451.eqiad.wmnet with reason: REIMAGE [production]
12:26 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1450.eqiad.wmnet with reason: REIMAGE [production]
12:26 <jelto@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1455.eqiad.wmnet with reason: REIMAGE [production]
12:24 <urbanecm> mwscript extensions/Translate/scripts/refresh-translatable-pages.php --wiki=commonswiki --jobqueue # T288683 [production]
12:24 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1455.eqiad.wmnet with reason: REIMAGE [production]
12:24 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1449.eqiad.wmnet with reason: REIMAGE [production]
12:22 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1448.eqiad.wmnet with reason: REIMAGE [production]
12:21 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1444.eqiad.wmnet [production]
12:21 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1449.eqiad.wmnet with reason: REIMAGE [production]
12:21 <mutante> mw1444 - scap pull, pooled as new API server for the first time [production]
12:20 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw1444.eqiad.wmnet [production]
12:19 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1448.eqiad.wmnet with reason: REIMAGE [production]
11:59 <urbanecm> mwscript extensions/Translate/scripts/refresh-translatable-pages.php --wiki=mediawikiwiki --jobqueue # T288683 [production]
11:36 <topranks> cloudsw1-d5-eqiad - configuring new 2x40G trunk to cloudsw2-d5-eqiad with homer (T277340) [production]
11:11 <jelto> mw1455 - powering on via mgmt - OS install, initial setup (T279309, T273915) [production]
10:22 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw1444.eqiad.wmnet with reason: new setup [production]
10:22 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mw1444.eqiad.wmnet with reason: new setup [production]
10:07 <filippo@puppetmaster1001> conftool action : set/pooled=yes; selector: name=thanos-fe2003.codfw.wmnet [production]
09:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw1444.eqiad.wmnet with reason: new setup [production]
09:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mw1444.eqiad.wmnet with reason: new setup [production]
09:42 <mutante> mw1448, mw1449, mw1450 - powering on via mgmt - OS install, initial setup (T279309, T273915) [production]
09:38 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on thanos-fe2003.codfw.wmnet with reason: REIMAGE [production]
09:35 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2003.codfw.wmnet with reason: REIMAGE [production]
09:35 <mutante> mw1444 - signed puppet cert, initial run (after hardware fix) T279309 [production]
09:17 <filippo@puppetmaster1001> conftool action : set/pooled=no; selector: name=thanos-fe2003.codfw.wmnet [production]
09:17 <filippo@puppetmaster1001> conftool action : set/pooled=yes; selector: name=thanos-fe2001.codfw.wmnet [production]
09:15 <filippo@puppetmaster1001> conftool action : set/pooled=yes; selector: name=thanos-fe2002.codfw.wmnet [production]
08:42 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2002.codfw.wmnet with reason: REIMAGE [production]
08:40 <filippo@puppetmaster1001> conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet [production]
08:40 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2002.codfw.wmnet with reason: REIMAGE [production]
05:24 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db1132.eqiad.wmnet with reason: REIMAGE [production]
05:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1132.eqiad.wmnet with reason: REIMAGE [production]