|
2019-06-19
ยง
|
| 14:11 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 14:06 |
<moritzm> |
rolling reboot of mwdebug servers for kernel security update |
[production] |
| 14:06 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:56 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Disabling Avro ApiAction Monolog channel - T222267 (duration: 00m 57s) |
[production] |
| 13:53 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 13:51 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 13:50 |
<cdanis> |
rebooting wikitech-static |
[production] |
| 13:48 |
<cdanis> |
apt upgrade on wikitech-static |
[production] |
| 13:47 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 13:44 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 13:27 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 13:24 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 13:20 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 13:17 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 13:00 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 12:57 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 12:53 |
<marostegui> |
Deploy schema change on the private wikis listed at T225643 |
[production] |
| 12:51 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 12:51 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 12:31 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 12:31 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 12:25 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 12:21 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 12:20 |
<ema> |
cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 T225998 |
[production] |
| 11:07 |
<ema> |
cache nodes: pause rolling reboots for kernel and varnish upgrades T224694 T225998 |
[production] |
| 10:59 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 10:54 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 10:52 |
<moritzm> |
rebooting mx1001 for kernel security update |
[production] |
| 10:50 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 10:47 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 10:38 |
<ladsgroup@deploy1001> |
scap-helm termbox finished |
[production] |
| 10:38 |
<ladsgroup@deploy1001> |
scap-helm termbox cluster codfw completed |
[production] |
| 10:38 |
<ladsgroup@deploy1001> |
scap-helm termbox upgrade -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: codfw] |
[production] |
| 10:36 |
<moritzm> |
rebooting mx2001 for kernel security update |
[production] |
| 10:35 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:33 |
<akosiaris@deploy1001> |
scap-helm termbox finished |
[production] |
| 10:33 |
<akosiaris@deploy1001> |
scap-helm termbox cluster staging completed |
[production] |
| 10:33 |
<akosiaris@deploy1001> |
scap-helm termbox upgrade -f termbox-staging-values.yaml staging stable/termbox [namespace: termbox, clusters: staging] |
[production] |
| 10:30 |
<jbond42> |
update late-install so it installs the correct puppet version https://gerrit.wikimedia.org/r/c/operations/puppet/+/515087 |
[production] |
| 10:30 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) |
[production] |
| 10:30 |
<moritzm> |
installing glibc and ca-certificates-java updates from stretch point release |
[production] |
| 10:29 |
<akosiaris@deploy1001> |
scap-helm termbox finished |
[production] |
| 10:29 |
<akosiaris@deploy1001> |
scap-helm termbox cluster eqiad completed |
[production] |
| 10:29 |
<akosiaris@deploy1001> |
scap-helm termbox upgrade -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad] |
[production] |
| 10:27 |
<ema@cumin1001> |
END (FAIL) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=99) |
[production] |
| 10:23 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 10:21 |
<ema@cumin1001> |
START - Cookbook sre.hosts.upgrade-and-reboot |
[production] |
| 10:05 |
<ema> |
cp3030: increase varnish-be thread_pool_max from 12000 (250 * 48) to 14400 (300 * 48) to observe impact on fetcherrors |
[production] |