1651-1700 of 10000 results (66ms)
2019-06-19 ยง
13:44 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
13:27 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
13:24 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
13:20 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
13:17 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
13:00 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:57 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:53 <marostegui> Deploy schema change on the private wikis listed at T225643 [production]
12:51 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:51 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:31 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:31 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:25 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:21 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:20 <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 T225998 [production]
11:07 <ema> cache nodes: pause rolling reboots for kernel and varnish upgrades T224694 T225998 [production]
10:59 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:54 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:52 <moritzm> rebooting mx1001 for kernel security update [production]
10:50 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:47 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:38 <ladsgroup@deploy1001> scap-helm termbox finished [production]
10:38 <ladsgroup@deploy1001> scap-helm termbox cluster codfw completed [production]
10:38 <ladsgroup@deploy1001> scap-helm termbox upgrade -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: codfw] [production]
10:36 <moritzm> rebooting mx2001 for kernel security update [production]
10:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:33 <akosiaris@deploy1001> scap-helm termbox finished [production]
10:33 <akosiaris@deploy1001> scap-helm termbox cluster staging completed [production]
10:33 <akosiaris@deploy1001> scap-helm termbox upgrade -f termbox-staging-values.yaml staging stable/termbox [namespace: termbox, clusters: staging] [production]
10:30 <jbond42> update late-install so it installs the correct puppet version https://gerrit.wikimedia.org/r/c/operations/puppet/+/515087 [production]
10:30 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:30 <moritzm> installing glibc and ca-certificates-java updates from stretch point release [production]
10:29 <akosiaris@deploy1001> scap-helm termbox finished [production]
10:29 <akosiaris@deploy1001> scap-helm termbox cluster eqiad completed [production]
10:29 <akosiaris@deploy1001> scap-helm termbox upgrade -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad] [production]
10:27 <ema@cumin1001> END (FAIL) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=99) [production]
10:23 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:21 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:05 <ema> cp3030: increase varnish-be thread_pool_max from 12000 (250 * 48) to 14400 (300 * 48) to observe impact on fetcherrors [production]
10:03 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1077 (duration: 00m 55s) [production]
10:01 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:56 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:54 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
09:49 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1077 (duration: 00m 55s) [production]
09:36 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:34 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
09:34 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1077 (duration: 00m 55s) [production]
09:29 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]