4001-4050 of 10000 results (71ms)
2022-08-10 ยง
15:02 <jelto> power off mc2035 [production]
15:01 <jelto> power off mc2034 [production]
15:01 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mc2035.codfw.wmnet with reason: PDU swap [production]
15:01 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mc2035.codfw.wmnet with reason: PDU swap [production]
15:01 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mc2034.codfw.wmnet with reason: PDU swap [production]
15:01 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on mc2034.codfw.wmnet with reason: PDU swap [production]
14:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: PDU Maint (T310146) [production]
14:43 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: PDU Maint (T310146) [production]
14:38 <urandom> disabling reserved space on eqiad nodes (RESTBase), /dev/md2 (aka /srv/cassandra/instance-data) -- T314941 [production]
14:28 <jelto> power off kafka-main2004 gracefully [production]
14:28 <hnowlan> shutting down sessionstore2003 [production]
14:27 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=sessionstore2003.codfw.wmnet [production]
14:27 <sukhe> power off cp2039, cp2040 for PDU upgrade: rack D [production]
14:27 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2003.codfw.wmnet with reason: PDU maintenance [production]
14:27 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2003.codfw.wmnet with reason: PDU maintenance [production]
14:25 <jelto> power off mc-gp2003 [production]
14:25 <jelto> power off mc2033 [production]
14:24 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:30:00 on kafka-main2004.codfw.wmnet with reason: PDU swap [production]
14:23 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 4:30:00 on kafka-main2004.codfw.wmnet with reason: PDU swap [production]
14:23 <sukhe> depool codfw for PDU upgrade: rack D [production]
14:23 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:30:00 on mc-gp2003.codfw.wmnet with reason: PDU swap [production]
14:23 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 4:30:00 on mc-gp2003.codfw.wmnet with reason: PDU swap [production]
14:23 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:30:00 on mc2033.codfw.wmnet with reason: PDU swap [production]
14:23 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 4:30:00 on mc2033.codfw.wmnet with reason: PDU swap [production]
14:15 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp20[39|40]\.codfw\.wmnet,service=ats-tls [production]
14:13 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp[2039-2040].codfw.wmnet with reason: shutdown for PDU upgrade: rack D4 [production]
14:13 <urandom> flushing Cassandra tables, restbase1030 [production]
14:13 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on cp[2039-2040].codfw.wmnet with reason: shutdown for PDU upgrade: rack D4 [production]
14:13 <urandom> flushing Cassandra tables, restbase1019 [production]
14:12 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dns2002.wikimedia.org with reason: shutdown for PDU upgrade: rack D4 [production]
14:12 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on dns2002.wikimedia.org with reason: shutdown for PDU upgrade: rack D4 [production]
14:11 <urandom> flushing Cassandra tables, restbase1017 1018 1021 1024 1025 1026 1028 1029 [production]
14:05 <urandom> flushing tables, restbase1016 [production]
13:52 <hnowlan> powered up restbase2018 [production]
13:32 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on netmon1003.wikimedia.org with reason: pdu [production]
13:32 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on netmon1003.wikimedia.org with reason: pdu [production]
13:32 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on logstash2003.codfw.wmnet with reason: pdu [production]
13:31 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on logstash2003.codfw.wmnet with reason: pdu [production]
13:31 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on logstash2029.codfw.wmnet with reason: pdu [production]
13:31 <root@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on logstash2029.codfw.wmnet with reason: pdu [production]
13:30 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
13:30 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
13:17 <elukey> powering on restbase2027 [production]
13:12 <elukey> powering on restbase2026 [production]
13:12 <_joe_> powering on restbase2023 [production]
13:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T312863)', diff saved to https://phabricator.wikimedia.org/P32343 and previous config saved to /var/cache/conftool/dbconfig/20220810-130108-ladsgroup.json [production]
13:01 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
13:00 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
12:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]
12:37 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]