7901-7950 of 10000 results (87ms)
2022-12-01 ยง
19:38 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1061'] [production]
19:35 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirt1061'] [production]
19:28 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1061'] [production]
19:28 <dancy@deploy1002> Finished scap: testing k8s deployment (duration: 06m 17s) [production]
19:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P42200 and previous config saved to /var/cache/conftool/dbconfig/20221201-192755-ladsgroup.json [production]
19:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
19:27 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirt1061'] [production]
19:27 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS buster [production]
19:25 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1061'] [production]
19:22 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1060.eqiad.wmnet with OS bullseye [production]
19:21 <dancy@deploy1002> Started scap: testing k8s deployment [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
19:16 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.12 refs T320517 [production]
19:15 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirt1061'] [production]
19:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
19:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P42199 and previous config saved to /var/cache/conftool/dbconfig/20221201-191248-ladsgroup.json [production]
19:09 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1057.eqiad.wmnet with OS bullseye [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
19:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1060.eqiad.wmnet with reason: host reimage [production]
19:02 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1060.eqiad.wmnet with reason: host reimage [production]
19:02 <dancy@deploy1002> Installation of scap version "4.30.0" completed for 601 hosts [production]
19:01 <dancy@deploy1002> Installing scap version "4.30.0" for 601 hosts [production]
18:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T323907)', diff saved to https://phabricator.wikimedia.org/P42197 and previous config saved to /var/cache/conftool/dbconfig/20221201-185742-ladsgroup.json [production]
18:55 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage [production]
18:51 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage [production]
18:43 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1061'] [production]
18:38 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudvirt1057.eqiad.wmnet with OS bullseye [production]
18:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudvirt1061'] [production]
18:37 <rzl@cumin2002> conftool action : set/pooled=no; selector: name=mw13(0[7-9]|[1-3]\d|4[0-8])\..* [production]
18:34 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1057.eqiad.wmnet with OS bullseye [production]
18:27 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/api-gateway: sync [production]
18:27 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/api-gateway: sync [production]
18:27 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/api-gateway: sync [production]
18:26 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/api-gateway: sync [production]
18:25 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/api-gateway: sync [production]
18:25 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/api-gateway: sync [production]
18:21 <bd808@deploy1002> helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [production]
18:19 <bd808@deploy1002> helmfile [eqiad] START helmfile.d/services/developer-portal: apply [production]
18:19 <bd808@deploy1002> helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [production]
18:17 <bd808@deploy1002> helmfile [codfw] START helmfile.d/services/developer-portal: apply [production]
18:17 <bd808@deploy1002> helmfile [staging] DONE helmfile.d/services/developer-portal: apply [production]
18:16 <bd808@deploy1002> helmfile [staging] START helmfile.d/services/developer-portal: apply [production]
18:16 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1059.eqiad.wmnet with OS bullseye [production]
18:14 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1061'] [production]
18:12 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudvirt1060.eqiad.wmnet with OS bullseye [production]
18:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1200 (T323907)', diff saved to https://phabricator.wikimedia.org/P42196 and previous config saved to /var/cache/conftool/dbconfig/20221201-181215-ladsgroup.json [production]