551-600 of 10000 results (37ms)
2021-03-23 ยง
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams-internal [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-main [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-logging-external [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics-external [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=echostore [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=cxserver [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=citoid [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=blubberoid [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=api-gateway [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=apertium [production]
08:33 <akosiaris> eqiad services in k8s depooled. T277741 [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=wikifeeds [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=termbox [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=similar-users [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=sessionstore [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=recommendation-api [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=push-notifications [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=proton [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mobileapps [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mathoid [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=linkrecommendation [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams-internal [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-main [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-logging-external [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics-external [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=echostore [production]
08:31 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=cxserver [production]
08:31 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=citoid [production]
08:31 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=blubberoid [production]
08:31 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=api-gateway [production]
08:31 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=apertium [production]
08:28 <akosiaris> downtime all services in T277741 for 24H [production]
08:25 <akosiaris> beginning the k8s upgrade/reinit process. T277741 [production]
08:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 75%: Slowly repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15022 and previous config saved to /var/cache/conftool/dbconfig/20210323-082454-root.json [production]
08:24 <moritzm> installing mariadb-10.3 updates on buster (just client-side libs/tools, unrelated to the main wmf-mariadb packages) [production]
08:24 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 18 hosts with reason: Reinitialize eqiad k8s cluster with new etcd [production]
08:24 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 18 hosts with reason: Reinitialize eqiad k8s cluster with new etcd [production]
08:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Slowly repool db1146:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P15021 and previous config saved to /var/cache/conftool/dbconfig/20210323-082213-root.json [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 50%: Slowly repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15020 and previous config saved to /var/cache/conftool/dbconfig/20210323-080949-root.json [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 75%: Slowly repool db1146:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P15019 and previous config saved to /var/cache/conftool/dbconfig/20210323-080709-root.json [production]
07:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 25%: Slowly repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15017 and previous config saved to /var/cache/conftool/dbconfig/20210323-075445-root.json [production]
07:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15016 and previous config saved to /var/cache/conftool/dbconfig/20210323-075253-marostegui.json [production]
07:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 100%: Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P15015 and previous config saved to /var/cache/conftool/dbconfig/20210323-075230-root.json [production]
07:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 100%: Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15014 and previous config saved to /var/cache/conftool/dbconfig/20210323-075216-root.json [production]
07:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 50%: Slowly repool db1146:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P15013 and previous config saved to /var/cache/conftool/dbconfig/20210323-075206-root.json [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 75%: Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P15012 and previous config saved to /var/cache/conftool/dbconfig/20210323-073726-root.json [production]