751-800 of 10000 results (105ms)
2024-11-04 ยง
19:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P70900 and previous config saved to /var/cache/conftool/dbconfig/20241104-193826-ladsgroup.json [production]
19:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T376905)', diff saved to https://phabricator.wikimedia.org/P70899 and previous config saved to /var/cache/conftool/dbconfig/20241104-192319-ladsgroup.json [production]
19:23 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
19:22 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
19:22 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [production]
19:21 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [production]
19:21 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [production]
19:20 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [production]
19:19 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [production]
19:18 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [production]
19:18 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
19:17 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
19:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1211 (T376905)', diff saved to https://phabricator.wikimedia.org/P70898 and previous config saved to /var/cache/conftool/dbconfig/20241104-191519-ladsgroup.json [production]
19:15 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:14 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70897 and previous config saved to /var/cache/conftool/dbconfig/20241104-191454-ladsgroup.json [production]
19:09 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:09 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:04 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:03 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P70896 and previous config saved to /var/cache/conftool/dbconfig/20241104-185947-ladsgroup.json [production]
18:58 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [production]
18:57 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-video: apply [production]
18:57 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [production]
18:55 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-media: apply [production]
18:55 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply [production]
18:54 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-constraints: apply [production]
18:54 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox: apply [production]
18:53 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox: apply [production]
18:47 <vgutierrez@cumin1002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: known issues with liberica-hcforwarder and ipip-multiqueue-optimizer [production]
18:47 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: known issues with liberica-hcforwarder and ipip-multiqueue-optimizer [production]
18:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P70895 and previous config saved to /var/cache/conftool/dbconfig/20241104-184440-ladsgroup.json [production]
18:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2013.codfw.wmnet [production]
18:41 <sukhe@cumin1002> START - Cookbook sre.hosts.remove-downtime for lvs2013.codfw.wmnet [production]
18:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: vgutierrez [production]
18:41 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: vgutierrez [production]
18:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70894 and previous config saved to /var/cache/conftool/dbconfig/20241104-182933-ladsgroup.json [production]
18:25 <vgutierrez@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1013.eqiad.wmnet with OS bookworm [production]
18:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70893 and previous config saved to /var/cache/conftool/dbconfig/20241104-182140-ladsgroup.json [production]
18:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
18:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
18:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T376905)', diff saved to https://phabricator.wikimedia.org/P70892 and previous config saved to /var/cache/conftool/dbconfig/20241104-182125-ladsgroup.json [production]
18:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P70891 and previous config saved to /var/cache/conftool/dbconfig/20241104-180618-ladsgroup.json [production]
18:01 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1013.eqiad.wmnet with reason: host reimage [production]
17:56 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1013.eqiad.wmnet with reason: host reimage [production]
17:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P70890 and previous config saved to /var/cache/conftool/dbconfig/20241104-175111-ladsgroup.json [production]