301-350 of 10000 results (22ms)
2021-06-25 §
13:06 <vgutierrez> upload trafficserver 8.0.8-1wm4 to apt.wm.o (buster) - T285535 [production]
12:28 <moritzm> installing nmap bugfix update from Buster point release [production]
12:28 <moritzm> installing nmal bugfix update from Buster point release [production]
11:28 <moritzm> installing 4.19.194 kernels on Buster from latest 10.10 point release (no reboots, just rolling out the packages) [production]
09:15 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:15 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:13 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:13 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:04 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2002.codfw.wmnet [production]
08:55 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet [production]
08:54 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet [production]
08:48 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet [production]
08:12 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet [production]
08:01 <elukey> reboot an-worker1101 to unblock stuck GPU [production]
08:00 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet [production]
08:00 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet [production]
07:57 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet [production]
07:42 <moritzm> imported Jenkins 2.289.1 to thirdparty/ci for buster-wikimedia T285531 [production]
07:30 <dcausse> depool and restart blazegraph on wdqs1005 [production]
07:17 <dcausse> installing openjdk-8-dbg on wdqs1005 to debug blazegraph [production]
2021-06-24 §
23:02 <legoktm> reverted cumin1001 spicerack live hacks [production]
22:57 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
22:36 <volans> set x2 codfw master back to RW [production]
22:30 <legoktm@cumin1001> END (ERROR) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=97) [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
22:29 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
22:29 <legoktm@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2021-06-24 22:29:25.643909 [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
22:09 <legoktm@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) [production]
22:09 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
22:06 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
22:05 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
22:04 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
22:04 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
22:01 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]