201-250 of 10000 results (47ms)
2022-09-06 ยง
11:17 <XioNoX> put cr4-ulsfo back in service - T295690 [production]
11:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1138 (re)pooling @ 5%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P33943 and previous config saved to /var/cache/conftool/dbconfig/20220906-111621-root.json [production]
11:12 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:12 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
11:12 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:11 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:11 <moritzm> installing ghostscript updates on stretch [production]
11:06 <XioNoX> restart cr4-ulsfo for software upgrade - T295690 [production]
11:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1138 (re)pooling @ 4%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P33942 and previous config saved to /var/cache/conftool/dbconfig/20220906-110116-root.json [production]
10:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33941 and previous config saved to /var/cache/conftool/dbconfig/20220906-105841-root.json [production]
10:58 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1009.eqiad.wmnet [production]
10:57 <cgoubert@cumin1001> START - Cookbook sre.hosts.remove-downtime for parse1009.eqiad.wmnet [production]
10:52 <moritzm> uploaded ghostscript 9.26a~dfsg-0+deb9u9+wmf1 to apt.wikimedia.org [production]
10:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1138 (re)pooling @ 3%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P33940 and previous config saved to /var/cache/conftool/dbconfig/20220906-104611-root.json [production]
10:44 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:44 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33939 and previous config saved to /var/cache/conftool/dbconfig/20220906-104336-root.json [production]
10:42 <XioNoX> drain traffic from cr4-ulsfo - T295690 [production]
10:40 <jayme> switched primary kube-controller-manager from kubemaster1001 to kubemaster1002 [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1188 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33938 and previous config saved to /var/cache/conftool/dbconfig/20220906-103402-root.json [production]
10:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1138 (re)pooling @ 2%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P33937 and previous config saved to /var/cache/conftool/dbconfig/20220906-103104-root.json [production]
10:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33936 and previous config saved to /var/cache/conftool/dbconfig/20220906-103017-root.json [production]
10:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1119 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33935 and previous config saved to /var/cache/conftool/dbconfig/20220906-102919-root.json [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33934 and previous config saved to /var/cache/conftool/dbconfig/20220906-102831-root.json [production]
10:27 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:27 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
10:26 <XioNoX> put cr3-ulsfo back in service - T295690 [production]
10:25 <btullis@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:25 <btullis@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
10:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1103 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33932 and previous config saved to /var/cache/conftool/dbconfig/20220906-102152-root.json [production]
10:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1188 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33931 and previous config saved to /var/cache/conftool/dbconfig/20220906-101858-root.json [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1138 (re)pooling @ 1%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P33930 and previous config saved to /var/cache/conftool/dbconfig/20220906-101559-root.json [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33929 and previous config saved to /var/cache/conftool/dbconfig/20220906-101513-root.json [production]
10:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1119 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33928 and previous config saved to /var/cache/conftool/dbconfig/20220906-101414-root.json [production]
10:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33927 and previous config saved to /var/cache/conftool/dbconfig/20220906-101326-root.json [production]
10:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2171:3316 (T314041)', diff saved to https://phabricator.wikimedia.org/P33926 and previous config saved to /var/cache/conftool/dbconfig/20220906-101129-ladsgroup.json [production]
10:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]
10:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]
10:06 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 100%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P33925 and previous config saved to /var/cache/conftool/dbconfig/20220906-100656-root.json [production]
10:06 <marostegui@cumin1001> dbctl commit (dc=all): 'db1103 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33924 and previous config saved to /var/cache/conftool/dbconfig/20220906-100647-root.json [production]
10:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1188 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33923 and previous config saved to /var/cache/conftool/dbconfig/20220906-100353-root.json [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33921 and previous config saved to /var/cache/conftool/dbconfig/20220906-100008-root.json [production]
09:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1119 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33920 and previous config saved to /var/cache/conftool/dbconfig/20220906-095909-root.json [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33919 and previous config saved to /var/cache/conftool/dbconfig/20220906-095821-root.json [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33918 and previous config saved to /var/cache/conftool/dbconfig/20220906-095722-root.json [production]
09:57 <cgoubert@puppetmaster1001> conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1009.eqiad.wmnet [production]
09:55 <claime> depooled wtp1041.eqiad.wmnet from parsoid cluster T307219 [production]
09:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 75%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P33917 and previous config saved to /var/cache/conftool/dbconfig/20220906-095151-root.json [production]
09:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1103 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33916 and previous config saved to /var/cache/conftool/dbconfig/20220906-095143-root.json [production]
09:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1188 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P33915 and previous config saved to /var/cache/conftool/dbconfig/20220906-094848-root.json [production]