2251-2300 of 10000 results (75ms)
2023-01-10 ยง
18:23 <jayme@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,cluster=kubernetes-staging,service=kubesvc [production]
18:21 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
18:20 <jayme@cumin1001> conftool action : set/pooled=inactive; selector: dc=codfw,cluster=kubernetes-staging,service=kubesvc [production]
18:19 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage [production]
18:16 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
18:16 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage [production]
18:09 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2001.codfw.wmnet with reason: host reimage [production]
18:06 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2001.codfw.wmnet with reason: host reimage [production]
18:01 <jayme@cumin1001> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bullseye [production]
18:01 <jayme@cumin1001> START - Cookbook sre.hosts.reimage for host kubestage2001.codfw.wmnet with OS bullseye [production]
17:55 <jayme@cumin1001> START - Cookbook sre.ganeti.reimage for host kubestagemaster2001.codfw.wmnet with OS bullseye [production]
17:51 <zabe> run populateCulActor on all wikis # T325484 [production]
17:48 <claime> Finished rolling reboots of eqiad appservers [production]
17:48 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
17:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
17:39 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
17:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1130 maint', diff saved to https://phabricator.wikimedia.org/P42956 and previous config saved to /var/cache/conftool/dbconfig/20230110-173807-ladsgroup.json [production]
17:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 T325652', diff saved to https://phabricator.wikimedia.org/P42955 and previous config saved to /var/cache/conftool/dbconfig/20230110-173027-marostegui.json [production]
17:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P42954 and previous config saved to /var/cache/conftool/dbconfig/20230110-173002-ladsgroup.json [production]
17:29 <ayounsi@deploy1002> Finished deploy [netbox/deploy@ef7451d]: netbox-next to 3.2.9 (duration: 00m 11s) [production]
17:28 <ayounsi@deploy1002> Started deploy [netbox/deploy@ef7451d]: netbox-next to 3.2.9 [production]
17:28 <ayounsi@deploy1002> deploy aborted: help (duration: 00m 01s) [production]
17:28 <ayounsi@deploy1002> Started deploy [netbox/deploy@ef7451d]: help [production]
17:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P42953 and previous config saved to /var/cache/conftool/dbconfig/20230110-171457-ladsgroup.json [production]
17:14 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
17:10 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
17:03 <ayounsi@deploy1002> deploy aborted: netbox-next to 3.2.9 (duration: 00m 07s) [production]
17:03 <ayounsi@deploy1002> Started deploy [netbox/deploy@ef7451d]: netbox-next to 3.2.9 [production]
16:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P42952 and previous config saved to /var/cache/conftool/dbconfig/20230110-165952-ladsgroup.json [production]
16:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 100%: After the incident', diff saved to https://phabricator.wikimedia.org/P42951 and previous config saved to /var/cache/conftool/dbconfig/20230110-165406-root.json [production]
16:48 <bblack> depooling eqsin from DNS [production]
16:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1130 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P42950 and previous config saved to /var/cache/conftool/dbconfig/20230110-164447-ladsgroup.json [production]
16:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: After the incident', diff saved to https://phabricator.wikimedia.org/P42949 and previous config saved to /var/cache/conftool/dbconfig/20230110-163901-root.json [production]
16:36 <jayme@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host kubestagetcd2003.codfw.wmnet with OS bullseye [production]
16:24 <jayme@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host kubestagetcd2001.codfw.wmnet with OS bullseye [production]
16:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: After the incident', diff saved to https://phabricator.wikimedia.org/P42948 and previous config saved to /var/cache/conftool/dbconfig/20230110-162356-root.json [production]
16:23 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagetcd2003.codfw.wmnet with reason: host reimage [production]
16:21 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagetcd2003.codfw.wmnet with reason: host reimage [production]
16:14 <jayme@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host kubestagetcd2002.codfw.wmnet with OS bullseye [production]
16:10 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: After the incident', diff saved to https://phabricator.wikimedia.org/P42947 and previous config saved to /var/cache/conftool/dbconfig/20230110-160851-root.json [production]
16:08 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:08 <jayme@cumin1001> START - Cookbook sre.ganeti.reimage for host kubestagetcd2003.codfw.wmnet with OS bullseye [production]
16:04 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagetcd2002.codfw.wmnet with reason: host reimage [production]
16:04 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version [production]
16:03 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on gitlab1004.wikimedia.org with reason: upgrade gitlab1004 to new version [production]
16:01 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagetcd2002.codfw.wmnet with reason: host reimage [production]
15:59 <SandraEbele> reran failed pageview-druid-hourly-coord oozie job for 2023-1-10-10. [production]
15:59 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:58 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]