151-200 of 10000 results (28ms)
2022-08-10 ยง
14:13 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on cp[2039-2040].codfw.wmnet with reason: shutdown for PDU upgrade: rack D4 [production]
14:13 <urandom> flushing Cassandra tables, restbase1019 [production]
14:12 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dns2002.wikimedia.org with reason: shutdown for PDU upgrade: rack D4 [production]
14:12 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on dns2002.wikimedia.org with reason: shutdown for PDU upgrade: rack D4 [production]
14:11 <urandom> flushing Cassandra tables, restbase1017 1018 1021 1024 1025 1026 1028 1029 [production]
14:05 <urandom> flushing tables, restbase1016 [production]
13:52 <hnowlan> powered up restbase2018 [production]
13:32 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on netmon1003.wikimedia.org with reason: pdu [production]
13:32 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on netmon1003.wikimedia.org with reason: pdu [production]
13:32 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on logstash2003.codfw.wmnet with reason: pdu [production]
13:31 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on logstash2003.codfw.wmnet with reason: pdu [production]
13:31 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on logstash2029.codfw.wmnet with reason: pdu [production]
13:31 <root@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on logstash2029.codfw.wmnet with reason: pdu [production]
13:30 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
13:30 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
13:17 <elukey> powering on restbase2027 [production]
13:12 <elukey> powering on restbase2026 [production]
13:12 <_joe_> powering on restbase2023 [production]
13:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T312863)', diff saved to https://phabricator.wikimedia.org/P32343 and previous config saved to /var/cache/conftool/dbconfig/20220810-130108-ladsgroup.json [production]
13:01 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
13:00 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
12:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]
12:37 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]
12:27 <jbond> remove confd from serveres that shouldn;t have it [production]
12:05 <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.23/extensions/Echo/maintenance/removeOrphanedEvents.php: Backport: [[gerrit:821735|Run clean ups with removeOrphanedEvents in major batches (T310428)]] (duration: 03m 32s) [production]
11:47 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
11:46 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
11:46 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
11:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
11:15 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS buster [production]
10:54 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
10:51 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
10:37 <jbond@cumin1001> START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS buster [production]
10:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2181-2182].codfw.wmnet with reason: D6 PDU maint (T310146) [production]
10:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db[2181-2182].codfw.wmnet with reason: D6 PDU maint (T310146) [production]
10:26 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on maps2010.codfw.wmnet with reason: PDU maintenance [production]
10:26 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on maps2010.codfw.wmnet with reason: PDU maintenance [production]
10:26 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps2010.codfw.wmnet [production]
10:25 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps2010.codfw.wmnet [production]
10:25 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps2008.codfw.wmnet [production]
10:24 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on restbase2018.codfw.wmnet with reason: PDU maintenance [production]
10:24 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase2018.codfw.wmnet [production]
10:24 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on restbase2018.codfw.wmnet with reason: PDU maintenance [production]
10:23 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ml-serve2008.codfw.wmnet with reason: PDU maintenance [production]
10:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on ml-serve2008.codfw.wmnet with reason: PDU maintenance [production]
10:20 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ores2009.codfw.wmnet with reason: PDU maintenance [production]
10:20 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on ores2009.codfw.wmnet with reason: PDU maintenance [production]
10:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ores2008.codfw.wmnet with reason: PDU maintenance [production]
10:19 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on ores2008.codfw.wmnet with reason: PDU maintenance [production]
10:03 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase202[367].codfw.wmnet [production]