1-50 of 10000 results (34ms)
2020-08-28 ยง
21:53 <ryankemper> `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues [production]
19:11 <andrewbogott> rebooting cloudvirt1006. It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving [production]
17:39 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
17:39 <mutante> shutting down mw2196 [production]
17:37 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
16:40 <rzl> switchdc live test complete [production]
16:36 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) [production]
16:35 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-update-tendril [production]
16:35 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
16:34 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
16:33 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
16:33 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
16:33 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
16:33 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
16:29 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
16:29 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period ends at: 2020-08-28 16:29:24.432463 [production]
16:29 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
16:29 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
16:29 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
16:29 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) [production]
16:29 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions [production]
16:29 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
16:29 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
16:28 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) [production]
16:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
16:28 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
16:28 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2020-08-28 16:28:07.882663 [production]
16:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
16:19 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
16:19 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
16:13 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
16:12 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
16:09 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
16:09 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
16:08 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
16:08 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
16:07 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
16:07 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
16:06 <rzl> starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise [production]
14:22 <moritzm> installing Java security updates on kafka/main and Logstash(5) clusters [production]
13:35 <hashar@deploy1001> Finished deploy [integration/docroot@65ec92c]: noop, sync up for README.md (duration: 00m 07s) [production]
13:35 <hashar@deploy1001> Started deploy [integration/docroot@65ec92c]: noop, sync up for README.md [production]
13:25 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:07 <elukey> stop kafka on kafka-jumbo1006 and reimage to buster [production]
12:56 <moritzm> installing debmonitor1002 T261492 [production]
12:46 <moritzm> installing debmonitor2002 T261492 [production]
11:50 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
11:40 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
11:37 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]