1201-1250 of 10000 results (37ms)
2021-03-23 ยง
15:32 <moritzm> installing libsdl2 security updates [production]
15:31 <akosiaris> pool echostore for eqiad (the first of the larger services traffic wise) [production]
15:31 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=echostore [production]
15:25 <Trey314159> reindexing Italian wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T274200) [production]
15:10 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
15:10 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
15:10 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
14:53 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
14:53 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
14:53 <akosiaris@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
14:46 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
14:46 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
14:46 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
14:43 <akosiaris> pool more services in eqiad k8s. T277741. Only the very large ones traffic wise are still on codfw [production]
14:43 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=recommendation-api [production]
14:43 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=push-notifications [production]
14:43 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=proton [production]
14:42 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=mobileapps [production]
14:42 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=mathoid [production]
14:42 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=linkrecommendation [production]
14:42 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=eventstreams-internal [production]
14:42 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=eventstreams [production]
14:20 <akosiaris> pool a few more services in eqiad k8s. T277741 [production]
14:19 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=wikifeeds [production]
14:19 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=termbox [production]
14:19 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=similar-users [production]
14:07 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.36 [production]
14:06 <akosiaris> pool a few services in eqiad k8s. T277741 [production]
14:05 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=cxserver [production]
14:05 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=citoid [production]
14:05 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=blubberoid [production]
14:05 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=api-gateway [production]
14:05 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=apertium [production]
14:05 <moritzm> installing pygments security updates on stretch [production]
14:04 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2008.codfw.wmnet [production]
13:59 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti2008.codfw.wmnet [production]
13:55 <hashar@deploy1002> Finished scap: Promote testwikis from 1.36.0-wmf.35 to 1.36.0-wmf.36 - T274940 (duration: 31m 57s) [production]
13:54 <elukey> sudo systemctl reload apache2 on prometheus[12]00[34] to pick up new k8s-mlserve instance settings [production]
13:28 <moritzm> drain ganeti2008 [production]
13:25 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet [production]
13:23 <hashar@deploy1002> Started scap: Promote testwikis from 1.36.0-wmf.35 to 1.36.0-wmf.36 - T274940 [production]
13:17 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet [production]
13:15 <ema> cp3054: install varnishkafka built explicitly against varnish 6.0.1-1wm2 to fix broken dpkg status T264398 [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1086 (re)pooling @ 100%: Slowly repool db1086 after cloning db1181', diff saved to https://phabricator.wikimedia.org/P15054 and previous config saved to /var/cache/conftool/dbconfig/20210323-130543-root.json [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1148 (re)pooling @ 100%: Slowly repool db1148 after schema change', diff saved to https://phabricator.wikimedia.org/P15053 and previous config saved to /var/cache/conftool/dbconfig/20210323-130153-root.json [production]
12:58 <moritzm> drain ganeti2018 [production]
12:58 <akosiaris> remove and decomission argon, chroline, acrab, acrux T277741, T277191 [production]
12:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 100%: Slowly pool db1165 into s6 T258361', diff saved to https://phabricator.wikimedia.org/P15052 and previous config saved to /var/cache/conftool/dbconfig/20210323-125155-root.json [production]
12:50 <marostegui@cumin1001> dbctl commit (dc=all): 'db1086 (re)pooling @ 75%: Slowly repool db1086 after cloning db1181', diff saved to https://phabricator.wikimedia.org/P15051 and previous config saved to /var/cache/conftool/dbconfig/20210323-125039-root.json [production]
12:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2017.codfw.wmnet [production]