3051-3100 of 10000 results (72ms)
2019-06-19 ยง
08:20 <akosiaris@deploy1001> scap-helm termbox finished [production]
08:20 <akosiaris@deploy1001> scap-helm termbox cluster staging completed [production]
08:20 <akosiaris@deploy1001> scap-helm termbox upgrade -f termbox-staging-values.yaml staging stable/termbox [namespace: termbox, clusters: staging] [production]
08:20 <akosiaris@deploy1001> scap-helm termbox finished [production]
08:20 <akosiaris@deploy1001> scap-helm termbox cluster codfw completed [production]
08:20 <akosiaris@deploy1001> scap-helm termbox cluster eqiad completed [production]
08:20 <akosiaris@deploy1001> scap-helm termbox upgrade -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad,codfw] [production]
08:19 <akosiaris@deploy1001> scap-helm mathoid finished [production]
08:18 <akosiaris@deploy1001> scap-helm mathoid cluster codfw completed [production]
08:18 <akosiaris@deploy1001> scap-helm mathoid cluster eqiad completed [production]
08:18 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-values.yaml production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] [production]
08:14 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
08:13 <akosiaris@deploy1001> scap-helm mathoid finished [production]
08:13 <akosiaris@deploy1001> scap-helm mathoid cluster staging completed [production]
08:13 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-staging-values.yaml staging stable/mathoid [namespace: mathoid, clusters: staging] [production]
08:13 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-staging-values.yaml staging stable/mathoid [namespace: mathoid, clusters: staging] [production]
08:13 <akosiaris@deploy1001> scap-helm citoid finished [production]
08:13 <akosiaris@deploy1001> scap-helm citoid cluster eqiad completed [production]
08:13 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-eqiad-values.yaml production stable/citoid [namespace: citoid, clusters: eqiad] [production]
08:08 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
08:07 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
08:02 <ema@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
08:01 <ema> cache nodes: resume rolling reboots for kernel and varnish upgrades T224694 [production]
08:00 <akosiaris@deploy1001> scap-helm citoid finished [production]
08:00 <akosiaris@deploy1001> scap-helm citoid cluster codfw completed [production]
08:00 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-codfw-values.yaml production stable/citoid [namespace: citoid, clusters: codfw] [production]
08:00 <akosiaris@deploy1001> scap-helm citoid finished [production]
07:59 <akosiaris@deploy1001> scap-helm citoid cluster staging completed [production]
07:59 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-staging-values.yaml staging stable/citoid [namespace: citoid, clusters: staging] [production]
07:56 <moritzm> rearmed keyholder on acmechief-test2001 [production]
07:51 <moritzm> installing vim security updates on stretch [production]
07:46 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-staging-values.yaml staging stable/mathoid [namespace: mathoid, clusters: staging] [production]
07:35 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-staging-values.yaml staging stable/mathoid [namespace: mathoid, clusters: staging] [production]
07:34 <akosiaris@deploy1001> scap-helm mathoid upgrade -f mathoid-staging-values.yaml staging stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] [production]
07:18 <XioNoX> jnt push to eqdfw, remove old protect-old-lvs-servers term + update syslog target T224128 [production]
07:17 <XioNoX> jnt push to eqord, remove old protect-old-lvs-servers term + update syslog target T224128 [production]
07:13 <XioNoX> jnt push to eqsin, remove old protect-old-lvs-servers term + update syslog target T224128 [production]
07:12 <marostegui> s3 will be lagging on labsdb hosts due to maintenance on db1077 - T225981 [production]
07:01 <XioNoX> jnt push to ulsfo, remove old protect-old-lvs-servers term + update syslog target T224128 [production]
06:57 <marostegui> Stop MySQL on db1077 to transfer its data to db1112 - T225981 [production]
06:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T225981 (duration: 01m 06s) [production]
05:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1135 T222682 (duration: 00m 56s) [production]
05:37 <marostegui> Upgrade db1068 (old s4 master) to 10.1.39 [production]
05:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1138 status (duration: 00m 55s) [production]
05:03 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove s4 ready only T224852 (duration: 00m 33s) [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Switchover s4 master eqiad from db1068 to db1081 T224852 (duration: 00m 33s) [production]
05:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set s4 on read-only T224852 (duration: 00m 34s) [production]
05:00 <marostegui> Starting s4 failover from db1068 to db1081 - T224852 [production]
04:40 <kartik@deploy1001> scap-helm cxserver finished [production]
04:40 <kartik@deploy1001> scap-helm cxserver cluster eqiad completed [production]