7151-7200 of 7789 results (13ms)
2022-08-22 §
13:38 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[1014-1016].eqiad.wmnet with reason: T314890 [production]
13:37 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[1014-1016].eqiad.wmnet with reason: T314890 [production]
13:37 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
13:31 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:31 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
13:25 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
2022-08-18 §
21:14 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: elastic 7 upgrade [production]
21:14 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: elastic 7 upgrade [production]
2022-08-16 §
21:53 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts elastic1048.eqiad.wmnet [production]
21:53 <bking@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:47 <bking@cumin1001> START - Cookbook sre.dns.netbox [production]
21:42 <bking@cumin1001> START - Cookbook sre.hosts.decommission for hosts elastic1048.eqiad.wmnet [production]
21:41 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts elastic1048.eqiad.wmnet [production]
21:31 <bking@cumin1001> START - Cookbook sre.hosts.decommission for hosts elastic1048.eqiad.wmnet [production]
2022-08-11 §
17:18 <bking@cumin1001> conftool action : set/weight=10:pooled=no; selector: service=elasticsearch-omega-ssl,name=elastic1100.eqiad.wmnet [production]
17:18 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: service=elasticsearch-omega-ssl,name=elastic1100.eqiad.wmnet [production]
17:15 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: service=search-omega-https,name=elastic1100.eqiad.wmnet [production]
16:29 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T309810 [production]
16:29 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T309810 [production]
16:26 <inflatador> bking@elastic1054 attempting to ban elastic1100-1102 from cluster due to firewall issues [production]
16:13 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: service=search-omega-https,name=elastic1100.eqiad.wmnet [production]
16:12 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=elastic1100 [production]
14:55 <inflatador> bking@cumin1001 running puppet agent across eqiad elastic hosts [production]
2022-08-10 §
21:25 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1016.eqiad.wmnet [production]
21:23 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1014.eqiad.wmnet [production]
21:10 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 16 hosts with reason: T309810 [production]
21:10 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 16 hosts with reason: T309810 [production]
21:09 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 [production]
21:09 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 [production]
20:09 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
18:47 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:30 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
13:30 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 6 hosts with reason: T310146 [production]
12:37 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]
12:37 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[2072,2084-2085].codfw.wmnet with reason: T310146 [production]
2022-08-09 §
23:17 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1011.eqiad.wmnet [production]
23:07 <bking@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
23:06 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
22:51 <bking@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
22:51 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
22:49 <bking@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
22:49 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
22:46 <bking@cumin1001> conftool action : set/weight=10:pooled=yes; selector: name=wdqs1015.eqiad.wmnet [production]
22:28 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
22:02 <bking@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
22:02 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
21:57 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2006.codfw.wmnet with reason: T310146 [production]
21:57 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2006.codfw.wmnet with reason: T310146 [production]
21:53 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
21:52 <bking@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]