1401-1450 of 10000 results (25ms)
2021-03-23 ยง
09:24 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1001.eqiad.wmnet with reason: REIMAGE [production]
09:18 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: dc=eqiad,cluster=kubernetes,name=kubernetes1017.eqiad.wmnet [production]
09:17 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,service=kubemaster,cluster=kubernetes [production]
09:17 <akosiaris@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,service=kubemaster,cluster=kubernetes [production]
09:16 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubernetes1017.eqiad.wmnet [production]
09:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1147', diff saved to https://phabricator.wikimedia.org/P15024 and previous config saved to /var/cache/conftool/dbconfig/20210323-091432-marostegui.json [production]
09:05 <akosiaris> reboot kubetcd100[456] for kernel upgrades. T277741 T273278 [production]
09:04 <akosiaris> empty etcd T277741 [production]
08:43 <akosiaris> poweroff argon and chlorine T277741 [production]
08:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 100%: Slowly repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P15023 and previous config saved to /var/cache/conftool/dbconfig/20210323-083957-root.json [production]
08:41 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=zotero [production]
08:41 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=wikifeeds [production]
08:41 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=termbox [production]
08:41 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=similar-users [production]
08:41 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=sessionstore [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=recommendation-api [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=push-notifications [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=proton [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mobileapps [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mathoid [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=linkrecommendation [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams-internal [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams [production]
08:40 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-main [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-logging-external [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics-external [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=echostore [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=cxserver [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=citoid [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=blubberoid [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=api-gateway [production]
08:39 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=apertium [production]
08:33 <akosiaris> eqiad services in k8s depooled. T277741 [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=wikifeeds [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=termbox [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=similar-users [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=sessionstore [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=recommendation-api [production]
08:33 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=push-notifications [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=proton [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mobileapps [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=mathoid [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=linkrecommendation [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams-internal [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventstreams [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-main [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-logging-external [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics-external [production]
08:32 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=eventgate-analytics [production]