2051-2100 of 10000 results (81ms)
2024-06-05 ยง
08:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P64062 and previous config saved to /var/cache/conftool/dbconfig/20240605-080247-marostegui.json [production]
08:01 <dcaro@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1026.eqiad.wmnet [production]
08:00 <dcaro@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1025.eqiad.wmnet [production]
08:00 <cgoubert@cumin1002> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw [production]
07:57 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mirror1001.wikimedia.org [production]
07:56 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage [production]
07:54 <dcaro@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1025.eqiad.wmnet [production]
07:53 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1186.eqiad.wmnet with reason: host reimage [production]
07:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host mirror1001.wikimedia.org [production]
07:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T364299)', diff saved to https://phabricator.wikimedia.org/P64061 and previous config saved to /var/cache/conftool/dbconfig/20240605-074739-marostegui.json [production]
07:45 <dcaro@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1021.eqiad.wmnet [production]
07:38 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS bookworm [production]
07:38 <dcaro@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1021.eqiad.wmnet [production]
07:38 <marostegui@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1186.eqiad.wmnet with OS bookworm [production]
07:38 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS bookworm [production]
07:37 <marostegui@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1186.eqiad.wmnet with OS bookworm [production]
07:37 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1186.eqiad.wmnet with OS bookworm [production]
07:35 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2004.wikimedia.org [production]
07:35 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1004.wikimedia.org [production]
07:31 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1186.eqiad.wmnet with reason: Reimage [production]
07:31 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host install2004.wikimedia.org [production]
07:30 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 5:00:00 on db1186.eqiad.wmnet with reason: Reimage [production]
07:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host install1004.wikimedia.org [production]
07:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1186', diff saved to https://phabricator.wikimedia.org/P64060 and previous config saved to /var/cache/conftool/dbconfig/20240605-073024-root.json [production]
07:28 <marostegui> dbmaint codfw s2 deploy schema change on db2207 T364299 [production]
07:27 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2207.codfw.wmnet with reason: Long schema change [production]
07:27 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 10:00:00 on db2207.codfw.wmnet with reason: Long schema change [production]
07:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2207 T366038', diff saved to https://phabricator.wikimedia.org/P64059 and previous config saved to /var/cache/conftool/dbconfig/20240605-072509-root.json [production]
07:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2204 to s2 primary T366038', diff saved to https://phabricator.wikimedia.org/P64058 and previous config saved to /var/cache/conftool/dbconfig/20240605-072427-marostegui.json [production]
07:24 <marostegui> Starting s2 codfw failover from db2207 to db2204 - T366038 [production]
07:08 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s2 T366038 [production]
07:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2204 with weight 0 T366038', diff saved to https://phabricator.wikimedia.org/P64057 and previous config saved to /var/cache/conftool/dbconfig/20240605-070758-root.json [production]
07:07 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s2 T366038 [production]
04:44 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1242 (T364069)', diff saved to https://phabricator.wikimedia.org/P64056 and previous config saved to /var/cache/conftool/dbconfig/20240605-044418-marostegui.json [production]
04:44 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [production]
04:43 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [production]
04:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241 (T364069)', diff saved to https://phabricator.wikimedia.org/P64055 and previous config saved to /var/cache/conftool/dbconfig/20240605-044355-marostegui.json [production]
04:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P64054 and previous config saved to /var/cache/conftool/dbconfig/20240605-042847-marostegui.json [production]
04:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P64053 and previous config saved to /var/cache/conftool/dbconfig/20240605-041339-marostegui.json [production]
04:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T364299)', diff saved to https://phabricator.wikimedia.org/P64052 and previous config saved to /var/cache/conftool/dbconfig/20240605-041306-marostegui.json [production]
04:12 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
04:12 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
04:12 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
04:12 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
04:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T364299)', diff saved to https://phabricator.wikimedia.org/P64051 and previous config saved to /var/cache/conftool/dbconfig/20240605-041227-marostegui.json [production]
03:58 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1177 (T352010)', diff saved to https://phabricator.wikimedia.org/P64050 and previous config saved to /var/cache/conftool/dbconfig/20240605-035855-ladsgroup.json [production]
03:58 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
03:58 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
03:58 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P64049 and previous config saved to /var/cache/conftool/dbconfig/20240605-035832-ladsgroup.json [production]
03:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241 (T364069)', diff saved to https://phabricator.wikimedia.org/P64048 and previous config saved to /var/cache/conftool/dbconfig/20240605-035831-marostegui.json [production]