|
2020-08-28
§
|
| 21:53 |
<ryankemper> |
`sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues |
[production] |
| 19:11 |
<andrewbogott> |
rebooting cloudvirt1006. It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving |
[production] |
| 17:39 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:39 |
<mutante> |
shutting down mw2196 |
[production] |
| 17:37 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 16:40 |
<rzl> |
switchdc live test complete |
[production] |
| 16:36 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |
| 16:35 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-update-tendril |
[production] |
| 16:35 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 16:34 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
[DRY-RUN] MediaWiki read-only period ends at: 2020-08-28 16:29:24.432463 |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki |
[production] |
| 16:28 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) |
[production] |
| 16:28 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly |
[production] |
| 16:28 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) |
[production] |
| 16:28 |
<rzl@cumin1001> |
[DRY-RUN] MediaWiki read-only period starts at: 2020-08-28 16:28:07.882663 |
[production] |
| 16:28 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly |
[production] |
| 16:19 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) |
[production] |
| 16:19 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance |
[production] |
| 16:13 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
| 16:12 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
| 16:09 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) |
[production] |
| 16:09 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
| 16:08 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) |
[production] |
| 16:08 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
| 16:07 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
| 16:07 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
| 16:06 |
<rzl> |
starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise |
[production] |
| 14:22 |
<moritzm> |
installing Java security updates on kafka/main and Logstash(5) clusters |
[production] |
| 13:35 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@65ec92c]: noop, sync up for README.md (duration: 00m 07s) |
[production] |
| 13:35 |
<hashar@deploy1001> |
Started deploy [integration/docroot@65ec92c]: noop, sync up for README.md |
[production] |
| 13:25 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:23 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:07 |
<elukey> |
stop kafka on kafka-jumbo1006 and reimage to buster |
[production] |
| 12:56 |
<moritzm> |
installing debmonitor1002 T261492 |
[production] |
| 12:46 |
<moritzm> |
installing debmonitor2002 T261492 |
[production] |
| 11:50 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 11:40 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |