551-600 of 10000 results (40ms)
2021-06-21 ยง
17:32 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs (duration: 04m 24s) [production]
17:27 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs [production]
16:47 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:32 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:46 <papaul> poweroff elastic2043 for maintenance [production]
15:25 <hashar> Updated operations-puppet-tests-buster-docker Jenkins job to use latest Docker image https://gerrit.wikimedia.org/r/c/integration/config/+/700648 [production]
15:06 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1009.eqiad.wmnet [production]
15:02 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1004.eqiad.wmnet [production]
15:01 <hnowlan@cumin1001> START - Cookbook sre.hosts.reboot-single for host maps1009.eqiad.wmnet [production]
14:57 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1004.eqiad.wmnet [production]
14:57 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1003.eqiad.wmnet [production]
14:52 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1003.eqiad.wmnet [production]
14:52 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1002.eqiad.wmnet [production]
14:47 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1002.eqiad.wmnet [production]
14:44 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1001.eqiad.wmnet [production]
14:40 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1001.eqiad.wmnet [production]
14:39 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet [production]
14:37 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet [production]
14:37 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet [production]
14:34 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet [production]
14:30 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet [production]
14:28 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet [production]
14:24 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet [production]
14:23 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1123.eqiad.wmnet with reason: REIMAGE [production]
14:22 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet [production]
14:21 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1123.eqiad.wmnet with reason: REIMAGE [production]
14:21 <volans> deployed spicerack release v0.0.54 on the cumin hosts [production]
14:19 <XioNoX> reboot scs-c1-codfw - T285229 [production]
14:18 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet [production]
14:17 <XioNoX> reboot scs-a1-codfw - T285229 [production]
14:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1008.eqiad.wmnet [production]
14:16 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet [production]
14:14 <klausman> starting update of ML team's etcd machines in eqiad [production]
14:14 <volans> uploaded spicerack_0.0.54 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
14:11 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2003.codfw.wmnet [production]
14:11 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps1008.eqiad.wmnet [production]
14:06 <otto@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
14:05 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2003.codfw.wmnet [production]
14:04 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2004.codfw.wmnet [production]
13:58 <XioNoX> reboot scs-eqsin - T285229 [production]
13:58 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2004.codfw.wmnet [production]
13:57 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet [production]
13:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1006.eqiad.wmnet [production]
13:56 <otto@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
13:55 <jynus> stopping replication at db1171:s3 at db1123-bin.004363:906878073 [production]
13:51 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet [production]
13:51 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2002.codfw.wmnet [production]
13:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host maps1006.eqiad.wmnet [production]
13:48 <XioNoX> reboot scs-ulsfo [production]
13:45 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve2002.codfw.wmnet [production]