|
2022-08-10
ยง
|
| 18:13 |
<urandom> |
truncating codfw Cassandra hints (eqiad datacenter) -- T314941 |
[production] |
| 18:07 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2005.codfw.wmnet |
[production] |
| 18:07 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for kafka-main2005.codfw.wmnet |
[production] |
| 18:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repool D8 DBs after PDU maint (T310146)', diff saved to https://phabricator.wikimedia.org/P32346 and previous config saved to /var/cache/conftool/dbconfig/20220810-180529-ladsgroup.json |
[production] |
| 17:42 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@6e47e0e]: Add missing changes to the deletion script - T270433 - [analytics/refinery@6e47e0e] (duration: 05m 28s) |
[production] |
| 17:39 |
<fnegri@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts labweb1002.wikimedia.org |
[production] |
| 17:39 |
<fnegri@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:36 |
<otto@deploy1002> |
Started deploy [analytics/refinery@6e47e0e]: Add missing changes to the deletion script - T270433 - [analytics/refinery@6e47e0e] |
[production] |
| 17:35 |
<fnegri@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:34 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@6e47e0e] (hadoop-test): Add missing changes to the deletion script - T270433 - TEST [analytics/refinery@6e47e0e] (duration: 04m 19s) |
[production] |
| 17:30 |
<fnegri@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts labweb1002.wikimedia.org |
[production] |
| 17:30 |
<otto@deploy1002> |
Started deploy [analytics/refinery@6e47e0e] (hadoop-test): Add missing changes to the deletion script - T270433 - TEST [analytics/refinery@6e47e0e] |
[production] |
| 17:09 |
<dzahn@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:08 |
<otto@deploy1002> |
Started deploy [analytics/refinery@d4dd7e4] (hadoop-test): Add safety limits to refinery-drop-older-than - T270433 - TEST [analytics/refinery@d4dd7e4] |
[production] |
| 17:06 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp4032: T309651 |
[production] |
| 17:06 |
<urandom> |
flushing RESTBase Cassandra tables -row B- to (temporarily) free instance-data space -- T314941 |
[production] |
| 17:05 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on krb2001.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 17:05 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on krb2001.codfw.wmnet with reason: btullis codfw maintenance |
[production] |
| 17:04 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts gerrit2001.wikimedia.org |
[production] |
| 17:02 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp6008: T309651 |
[production] |
| 16:56 |
<sukhe> |
testing ATS 9.1.3-1wm1 on cp6016: T309651 |
[production] |
| 16:55 |
<fnegri@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts labweb1001.wikimedia.org |
[production] |
| 16:55 |
<fnegri@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:32 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gerrit2001.wikimedia.org |
[production] |
| 16:32 |
<dzahn@cumin2002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 16:32 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kubernetes[2013-2014].codfw.wmnet |
[production] |
| 16:31 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for kubernetes[2013-2014].codfw.wmnet |
[production] |
| 16:31 |
<jelto> |
kubectl uncordon kubernetes2014.codfw.wmnet |
[production] |
| 16:31 |
<fnegri@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:30 |
<jelto> |
kubectl uncordon kubernetes2013.codfw.wmnet |
[production] |
| 16:29 |
<urandom> |
restarting Cassandra (RESTBase) -row A- to apply r822110 -- T314941 |
[production] |
| 16:27 |
<dzahn@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:25 |
<fnegri@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts labweb1001.wikimedia.org |
[production] |
| 16:23 |
<mutante> |
shutting down gerrit2001 |
[production] |
| 16:23 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc[2034-2035].codfw.wmnet |
[production] |
| 16:23 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for mc[2034-2035].codfw.wmnet |
[production] |
| 16:22 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2016-2018].codfw.wmnet |
[production] |
| 16:22 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for parse[2016-2018].codfw.wmnet |
[production] |
| 16:16 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=maps2010.codfw.wmnet |
[production] |
| 16:16 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=sessionstore2003.codfw.wmnet |
[production] |
| 16:13 |
<sukhe> |
reprepro -C component/trafficserver9 include buster-wikimedia trafficserver_9.1.3-1wm1_amd64.changes: T309651 |
[production] |
| 16:13 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts gerrit2001.wikimedia.org |
[production] |
| 16:11 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be[2039,2050,2056,2059].codfw.wmnet,thanos-be2004.codfw.wmnet with reason: PDU work |
[production] |
| 16:10 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be[2039,2050,2056,2059].codfw.wmnet,thanos-be2004.codfw.wmnet with reason: PDU work |
[production] |
| 16:09 |
<urandom> |
flushing tables in row D (RESTBase Cassandra cluster) -- T314941 |
[production] |
| 15:54 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for gitlab-runner2004.codfw.wmnet |
[production] |
| 15:54 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for gitlab-runner2004.codfw.wmnet |
[production] |
| 15:53 |
<sukhe> |
poweroff cp2041, 42 for PDU ugprade: rack D7 |
[production] |
| 15:51 |
<urandom> |
flushing tables in row B (RESTBase Cassandra cluster) -- T314941 |
[production] |
| 15:49 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: PDU maintenance |
[production] |