4001-4050 of 10000 results (83ms)
2022-12-15 ยง
13:06 <btullis@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" [production]
12:51 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-stretch1001.eqiad.wmnet with reason: host reimage [production]
12:48 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-stretch1001.eqiad.wmnet with reason: host reimage [production]
12:36 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-stretch1001.eqiad.wmnet with OS bullseye [production]
12:20 <jiji@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw [production]
12:19 <jiji@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=kartotherian,name=maps1010.eqiad.wmnet [production]
12:19 <jiji@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=kartotherian-ssl,name=maps1010.eqiad.wmnet [production]
12:12 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1002.wikimedia.org [production]
12:08 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@00c9a16] (eqiad): codfw: Disable traffic mirroring (duration: 01m 00s) [production]
12:07 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@00c9a16] (eqiad): codfw: Disable traffic mirroring [production]
12:07 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netmon1002.wikimedia.org [production]
11:58 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host seaborgium.wikimedia.org [production]
11:54 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org [production]
11:43 <jiji@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [production]
11:42 <effie> switching maps/kartotherian from codfw to eqiad [production]
11:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1001.eqiad.wmnet [production]
11:39 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@00c9a16] (codfw): codfw: Disable traffic mirroring (duration: 01m 43s) [production]
11:37 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@00c9a16] (codfw): codfw: Disable traffic mirroring [production]
11:34 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host krb1001.eqiad.wmnet [production]
11:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flowspec1001.eqiad.wmnet [production]
11:21 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host flowspec1001.eqiad.wmnet [production]
11:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping3002.esams.wmnet [production]
11:15 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test-eqiad cluster: Reboot kafka nodes [production]
11:11 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ping3002.esams.wmnet [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2002.codfw.wmnet [production]
11:00 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ping2002.codfw.wmnet [production]
10:53 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1002.eqiad.wmnet [production]
10:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ping1002.eqiad.wmnet [production]
10:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2001.codfw.wmnet [production]
10:47 <XioNoX> disable ping offload in eqiad [production]
10:43 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host krb2001.codfw.wmnet [production]
10:42 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2001.wikimedia.org [production]
10:38 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host urldownloader2001.wikimedia.org [production]
10:36 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1002.wikimedia.org [production]
10:34 <jayme> restarted istiod pods in aux-k8s because of T303184 [production]
10:32 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host urldownloader1002.wikimedia.org [production]
09:56 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1001.eqiad.wmnet [production]
09:54 <effie> stopping and masking nutcracker on mw servers - T277183 [production]
09:53 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host acmechief1001.eqiad.wmnet [production]
09:51 <vgutierrez@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host acmechief2001.codfw.wmnet [production]
09:42 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org [production]
09:41 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host acmechief2001.codfw.wmnet [production]
09:40 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet [production]
09:38 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org [production]
09:38 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet [production]
09:37 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet [production]
09:31 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet [production]
09:30 <vgutierrez@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host acmechief-test1001.eqiad.wmnet [production]
09:30 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet [production]
09:27 <elukey@cumin1001> START - Cookbook sre.kafka.reboot-workers for Kafka test-eqiad cluster: Reboot kafka nodes [production]