3801-3850 of 10000 results (27ms)
2021-03-02 ยง
17:39 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/Graph/: Do not log graph errors to WMF servers (duration: 01m 08s) [production]
17:21 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.32/extensions/ContentTranslation/: Re-apply: CX3 Build 0.1.0+20210223 (duration: 01m 10s) [production]
16:37 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=eqiad [production]
16:37 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet [production]
16:33 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet [production]
16:14 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
16:13 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
16:10 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
16:10 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
15:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 100%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14563 and previous config saved to /var/cache/conftool/dbconfig/20210302-155932-root.json [production]
15:56 <jayme@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=eqiad [production]
15:56 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=codfw [production]
15:55 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet [production]
15:52 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet [production]
15:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 85%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14562 and previous config saved to /var/cache/conftool/dbconfig/20210302-154429-root.json [production]
15:35 <tgr@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/GrowthExperiments/: Backport: [[gerrit:667811|HomepageHooks: Block search data hook if link recommendations are off (T276224)]] (duration: 01m 13s) [production]
15:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 75%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14561 and previous config saved to /var/cache/conftool/dbconfig/20210302-152925-root.json [production]
15:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 50%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14560 and previous config saved to /var/cache/conftool/dbconfig/20210302-151422-root.json [production]
15:00 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:00 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 25%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14559 and previous config saved to /var/cache/conftool/dbconfig/20210302-145918-root.json [production]
14:57 <vgutierrez> rolling restart of ats-tls on codfw [production]
14:53 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
14:53 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 10%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14558 and previous config saved to /var/cache/conftool/dbconfig/20210302-144415-root.json [production]
14:43 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:43 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:29 <jynus> dropping db grants for bacula from m1 T274809 [production]
14:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1084 (re)pooling @ 5%: Repool db1084 after cloning db1164', diff saved to https://phabricator.wikimedia.org/P14557 and previous config saved to /var/cache/conftool/dbconfig/20210302-142911-root.json [production]
14:07 <jynus> dropping database bacula from m1 (with replication) T274809 [production]
14:04 <liw@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.33 [production]
13:57 <moritzm> installing bind9 security updates on stretch (client-side tools/libs only) [production]
13:56 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubestagemaster1001.eqiad.wmnet [production]
13:44 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubemaster1002.eqiad.wmnet [production]
13:42 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm for new host kubestagemaster1001.eqiad.wmnet [production]
13:25 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm for new host kubemaster1002.eqiad.wmnet [production]
13:24 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubemaster1001.eqiad.wmnet [production]
13:16 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubemaster2001.codfw.wmnet [production]
13:13 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
13:10 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
13:08 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm for new host kubemaster1001.eqiad.wmnet [production]
12:53 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubemaster2002.codfw.wmnet [production]
12:53 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
12:46 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm for new host kubemaster2001.codfw.wmnet [production]
12:44 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1012.eqiad.wmnet [production]
12:43 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host kubemaster2001.codfw.wmnet [production]
12:39 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1012.eqiad.wmnet [production]
12:32 <jayme@cumin1001> END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) [production]
12:32 <jayme@cumin1001> START - Cookbook sre.discovery.service-route [production]