|
2022-08-10
ยง
|
| 17:06 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp4032: T309651 |
[production] |
| 17:06 |
<urandom> |
flushing RESTBase Cassandra tables -row B- to (temporarily) free instance-data space -- T314941 |
[production] |
| 17:05 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on krb2001.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 17:05 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on krb2001.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 17:04 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts gerrit2001.wikimedia.org |
[production] |
| 17:02 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp6008: T309651 |
[production] |
| 16:56 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp6016: T309651 |
[production] |
| 16:55 |
<fnegri@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts labweb1001.wikimedia.org |
[production] |
| 16:55 |
<fnegri@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:32 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gerrit2001.wikimedia.org |
[production] |
| 16:32 |
<dzahn@cumin2002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 16:32 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kubernetes[2013-2014].codfw.wmnet |
[production] |
| 16:31 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for kubernetes[2013-2014].codfw.wmnet |
[production] |
| 16:31 |
<jelto> |
kubectl uncordon kubernetes2014.codfw.wmnet |
[production] |
| 16:31 |
<fnegri@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:30 |
<jelto> |
kubectl uncordon kubernetes2013.codfw.wmnet |
[production] |
| 16:29 |
<urandom> |
restarting Cassandra (RESTBase) -row A- to apply r822110 -- T314941 |
[production] |
| 16:27 |
<dzahn@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:25 |
<fnegri@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts labweb1001.wikimedia.org |
[production] |
| 16:23 |
<mutante> |
shutting down gerrit2001 |
[production] |
| 16:23 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc[2034-2035].codfw.wmnet |
[production] |
| 16:23 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for mc[2034-2035].codfw.wmnet |
[production] |
| 16:22 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2016-2018].codfw.wmnet |
[production] |
| 16:22 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for parse[2016-2018].codfw.wmnet |
[production] |
| 16:16 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=maps2010.codfw.wmnet |
[production] |
| 16:16 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=sessionstore2003.codfw.wmnet |
[production] |
| 16:13 |
<sukhe> |
reprepro -C component/trafficserver9 include buster-wikimedia trafficserver_9.1.3-1wm1_amd64.changes: T309651 |
[production] |
| 16:13 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts gerrit2001.wikimedia.org |
[production] |
| 16:11 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be[2039,2050,2056,2059].codfw.wmnet,thanos-be2004.codfw.wmnet with reason: PDU work |
[production] |
| 16:10 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be[2039,2050,2056,2059].codfw.wmnet,thanos-be2004.codfw.wmnet with reason: PDU work |
[production] |
| 16:09 |
<urandom> |
flushing tables in row D (RESTBase Cassandra cluster) -- T314941 |
[production] |
| 15:54 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for gitlab-runner2004.codfw.wmnet |
[production] |
| 15:54 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for gitlab-runner2004.codfw.wmnet |
[production] |
| 15:53 |
<sukhe> |
poweroff cp2041, 42 for PDU ugprade: rack D7 |
[production] |
| 15:51 |
<urandom> |
flushing tables in row B (RESTBase Cassandra cluster) -- T314941 |
[production] |
| 15:49 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: PDU maintenance |
[production] |
| 15:49 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: PDU maintenance |
[production] |
| 15:46 |
<urandom> |
flushing tables in row A (RESTBase Cassandra cluster) -- T314941 |
[production] |
| 15:46 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on aqs2012.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:46 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp[2041-2042].codfw.wmnet with reason: shutdown for PDU upgrade: rack D4 |
[production] |
| 15:46 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on aqs2012.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:46 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on cp[2041-2042].codfw.wmnet with reason: shutdown for PDU upgrade: rack D4 |
[production] |
| 15:46 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on aqs2011.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:46 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on aqs2011.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:45 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on aqs2010.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:45 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on aqs2010.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:45 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on aqs2009.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:45 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on aqs2009.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 15:37 |
<urandom> |
(ephemerally) increasing hinted hand-off delivery rate limit to 16KB, RESTBase eqiad nodes -- T314941 |
[production] |
| 15:34 |
<jbond> |
remove puppetmaster[12]002 from production |
[production] |