101-150 of 10000 results (89ms)
2024-11-04 ยง
20:08 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T376905)', diff saved to https://phabricator.wikimedia.org/P70902 and previous config saved to /var/cache/conftool/dbconfig/20241104-200840-ladsgroup.json [production]
20:00 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1087231|Message: Downgrade exception on bool/null param to warning (T378876)]] (duration: 09m 12s) [production]
19:55 <urbanecm@deploy2002> urbanecm: Continuing with sync [production]
19:54 <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:1087231|Message: Downgrade exception on bool/null param to warning (T378876)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
19:53 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P70901 and previous config saved to /var/cache/conftool/dbconfig/20241104-195333-ladsgroup.json [production]
19:51 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1087231|Message: Downgrade exception on bool/null param to warning (T378876)]] [production]
19:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P70900 and previous config saved to /var/cache/conftool/dbconfig/20241104-193826-ladsgroup.json [production]
19:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T376905)', diff saved to https://phabricator.wikimedia.org/P70899 and previous config saved to /var/cache/conftool/dbconfig/20241104-192319-ladsgroup.json [production]
19:23 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
19:22 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
19:22 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [production]
19:21 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [production]
19:21 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [production]
19:20 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [production]
19:19 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [production]
19:18 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [production]
19:18 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
19:17 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
19:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1211 (T376905)', diff saved to https://phabricator.wikimedia.org/P70898 and previous config saved to /var/cache/conftool/dbconfig/20241104-191519-ladsgroup.json [production]
19:15 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:14 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70897 and previous config saved to /var/cache/conftool/dbconfig/20241104-191454-ladsgroup.json [production]
19:09 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:09 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:04 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
19:03 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P70896 and previous config saved to /var/cache/conftool/dbconfig/20241104-185947-ladsgroup.json [production]
18:58 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [production]
18:57 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-video: apply [production]
18:57 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
18:56 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [production]
18:55 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-media: apply [production]
18:55 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply [production]
18:54 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-constraints: apply [production]
18:54 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox: apply [production]
18:53 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox: apply [production]
18:47 <vgutierrez@cumin1002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: known issues with liberica-hcforwarder and ipip-multiqueue-optimizer [production]
18:47 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: known issues with liberica-hcforwarder and ipip-multiqueue-optimizer [production]
18:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P70895 and previous config saved to /var/cache/conftool/dbconfig/20241104-184440-ladsgroup.json [production]
18:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2013.codfw.wmnet [production]
18:41 <sukhe@cumin1002> START - Cookbook sre.hosts.remove-downtime for lvs2013.codfw.wmnet [production]
18:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: vgutierrez [production]
18:41 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on lvs2013.codfw.wmnet with reason: vgutierrez [production]
18:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70894 and previous config saved to /var/cache/conftool/dbconfig/20241104-182933-ladsgroup.json [production]
18:25 <vgutierrez@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1013.eqiad.wmnet with OS bookworm [production]
18:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1209 (T376905)', diff saved to https://phabricator.wikimedia.org/P70893 and previous config saved to /var/cache/conftool/dbconfig/20241104-182140-ladsgroup.json [production]
18:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]