501-550 of 10000 results (23ms)
2021-07-20 ยง
15:23 <vgutierrez> pool dns1002 - T286069 [production]
15:21 <vgutierrez> pool cp[1087-1090].eqiad.wmnet - T286069 [production]
15:19 <jmm@puppetmaster1001> conftool action : set/pooled=yes; selector: name=ldap-replica1004.wikimedia.org [production]
15:14 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1297.eqiad.wmnet [production]
15:14 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1290.eqiad.wmnet [production]
15:14 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1289.eqiad.wmnet [production]
15:06 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: Deploying schema change to s3 T281058 [production]
15:06 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 12 hosts with reason: Deploying schema change to s3 T281058 [production]
14:53 <urbanecm> Start server-side upload for 7 large PNG files (T285708) [production]
14:51 <herron> depooled and scheduled downtime for kafka-main100[45] [production]
14:51 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs1016.eqiad.wmnet with reason: eqiad row D maintenance [production]
14:50 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs1016.eqiad.wmnet with reason: eqiad row D maintenance [production]
14:48 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dns1002.wikimedia.org with reason: eqiad row D maintenance [production]
14:48 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on dns1002.wikimedia.org with reason: eqiad row D maintenance [production]
14:46 <vgutierrez> depool dns1002 - T286069 [production]
14:40 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp[1087-1090].eqiad.wmnet with reason: eqiad row D maintenance [production]
14:40 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on cp[1087-1090].eqiad.wmnet with reason: eqiad row D maintenance [production]
14:36 <vgutierrez> depool cp[1087-1090].eqiad.wmnet - T286069 [production]
14:30 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 18 hosts with reason: Deploying schema change to s8 T281058 [production]
14:30 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 18 hosts with reason: Deploying schema change to s8 T281058 [production]
14:25 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [production]
14:25 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [production]
14:22 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
14:21 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
14:12 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 18 hosts with reason: Deploying schema change to s4 T281058 [production]
14:12 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 18 hosts with reason: Deploying schema change to s4 T281058 [production]
14:09 <jiji@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
14:08 <jiji@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
14:03 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps2009.codfw.wmnet [production]
14:00 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
13:56 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps2008.codfw.wmnet [production]
13:50 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Deploying schema change to s7 T281058 [production]
13:50 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Deploying schema change to s7 T281058 [production]
13:45 <hnowlan@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [production]
13:45 <hnowlan@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw [production]
13:43 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps200[89].codfw.wmnet [production]
13:30 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps20(10|0[1-9]).codfw.wmnet [production]
13:25 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Deploying schema change to s2 T281058 [production]
13:25 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Deploying schema change to s2 T281058 [production]
13:14 <gehel> set/pooled=inactive on elastic1039 - disk failure - T285643 [production]
13:14 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 13 hosts with reason: Deploying schema change to s5 T281058 [production]
13:14 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 13 hosts with reason: Deploying schema change to s5 T281058 [production]
13:13 <gehel@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=elastic1039.eqiad.wmnet [production]
12:44 <moritzm> installing systemd security updates on buster [production]
12:23 <elukey> reboot ml-serve-ctrl vms to pick up new vcores settings [production]
12:22 <elukey> bump vcpus from 2 to 4 on ml-serve-ctrl VMs on Ganeti (load/cpu usage increased steadily since we deployed kubelets on them) [production]
11:58 <Lucas_WMDE> EU config+backport window done [production]
11:58 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/CommonSettings-labs.php: Config: [[gerrit:705505|Avoid using User::newFrom* methods]] (3/3) (duration: 00m 56s) [production]
11:58 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on maps1007.eqiad.wmnet with reason: Testing impact of tilerator [production]
11:58 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on maps1007.eqiad.wmnet with reason: Testing impact of tilerator [production]