301-350 of 10000 results (52ms)
2020-02-12 ยง
16:51 <bblack@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:49 <moritzm> installing openjpeg2 security updates [production]
16:10 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:08 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:07 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:05 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:56 <vgutierrez> Enable KA and disable parent proxies on cp4031 - T244464 [production]
15:50 <vgutierrez> depool cp20[06,14] and reimage as buster - T242093 [production]
15:49 <volans> spicerack upgraded to 0.0.30-1 on both cumin hosts [production]
15:48 <vgutierrez> pool cp20[07,17] running buster - T242093 [production]
15:46 <bblack> authdns2001 - shutting down for hardware work - T242017 [production]
15:40 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:39 <jeh> clearing foreign drive RAID configuration on cloudvirt1024 T241884 [production]
15:37 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:32 <marostegui> Disable event handler for db1095 RAID check on icinga - T244958 [production]
15:32 <marostegui> Disable event handler for db1095 RAID check on icinga - [production]
15:28 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:24 <jeh> upgrade BIOS firmware on cloudvirt1024 to 2.4.8 T241884 [production]
15:19 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:17 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <vgutierrez> depool cp20[07,17] and reimage as buster - T242093 [production]
14:34 <XioNoX> repool eqsin [production]
14:31 <moritzm> reimage logstash2026 to test new standard RAID0 partman recipe [production]
14:00 <vgutierrez> pool cp20[10,18] running buster - T242093 [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10393 and previous config saved to /var/cache/conftool/dbconfig/20200212-135514-marostegui.json [production]
13:39 <akosiaris> revert sessionstore on mw1331, mw1348 so that it times out instead of returning TCP RSTs. Testing for T243106 [production]
13:36 <XioNoX> re-enable transit/peering on cr1-eqsin - T244944 [production]
13:26 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:24 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:23 <akosiaris> mangle sessionstore on mw1331, mw1348 so that it timesout instead of returning TCP RSTs. Testing for T243106 [production]
13:23 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:22 <XioNoX> cr1-eqsin RE failover (final) - T244944 [production]
13:21 <marostegui> Restart wikibugs as phab comments aren't showing up on irc - T241109 [production]
13:20 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:18 <jynus> setting up db1140 under maintenance (upgrade, reboot, disable alerts) [production]
13:15 <vgutierrez> disabling KA between ats-tls and varnish-fe on cp4031 - T244464 [production]
13:10 <moritzm> upgrading debdeploy fleet-wide to 0.0.99.13 [production]
13:08 <moritzm> uploaded libapache2-mod-auth-cas 1.2-1~deb8u1 for jessie-wikimedia to apt.wikimedia.org [production]
13:05 <vgutierrez> depool cp20[10,18] and reimage as buster - T242093 [production]
13:05 <vgutierrez> pool cp20[12,20] running buster - T242093 [production]
12:55 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:53 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:53 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:53 <XioNoX> cr1-eqsin RE failover - T244944 [production]
12:50 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:35 <vgutierrez> depool cp20[12,20] and reimage as buster - T242093 [production]
12:34 <vgutierrez> pool cp20[13,22] running buster - T242093 [production]
12:26 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:24 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]