6001-6050 of 10000 results (66ms)
2022-06-08 ยง
17:43 <rzl> rolled back shellbox main to revision 2 on eqiad, to unstick a stuck upgrade [production]
17:38 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:38 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:36 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti3003.esams.wmnet with reason: host reimage [production]
17:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T310011)', diff saved to https://phabricator.wikimedia.org/P29558 and previous config saved to /var/cache/conftool/dbconfig/20220608-173536-marostegui.json [production]
17:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti3003.esams.wmnet with reason: host reimage [production]
17:32 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
17:32 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:29 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
17:29 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
17:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:24 <hashar@deploy1002> Finished deploy [integration/docroot@e810fc7]: Update Wikibase section (duration: 00m 08s) [production]
17:24 <hashar@deploy1002> Started deploy [integration/docroot@e810fc7]: Update Wikibase section [production]
17:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1156 (T310011)', diff saved to https://phabricator.wikimedia.org/P29556 and previous config saved to /var/cache/conftool/dbconfig/20220608-172305-marostegui.json [production]
17:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
17:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
17:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]
17:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]
17:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122 (T310011)', diff saved to https://phabricator.wikimedia.org/P29555 and previous config saved to /var/cache/conftool/dbconfig/20220608-172252-marostegui.json [production]
17:22 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/shellbox: apply [production]
17:21 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox: apply [production]
17:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:15 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host ganeti3003.esams.wmnet with OS bullseye [production]
17:14 <krinkle@deploy1002> Synchronized src/Profiler.php: I534fb954c359c29a3f018eec75f62b4c4bfcd23f (duration: 03m 35s) [production]
17:13 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/shellbox: apply [production]
17:13 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/shellbox: apply [production]
17:13 <rzl> the above "helmfile -i apply" was canceled [production]
17:12 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/shellbox: apply [production]
17:11 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox: apply [production]
17:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P29554 and previous config saved to /var/cache/conftool/dbconfig/20220608-170747-marostegui.json [production]
17:06 <dancy@deploy1002> prep aborted: (duration: 24m 59s) [production]
17:02 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2050.codfw.wmnet [production]
16:57 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2050.codfw.wmnet [production]
16:57 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2049.codfw.wmnet [production]
16:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P29553 and previous config saved to /var/cache/conftool/dbconfig/20220608-165242-marostegui.json [production]
16:51 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2049.codfw.wmnet [production]
16:51 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2048.codfw.wmnet [production]
16:46 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2048.codfw.wmnet [production]
16:46 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2047.codfw.wmnet [production]
16:44 <robh> ganeti3003 (already depooled) coming down for firmware update and reimage via T308238 [production]
16:44 <robh> ganeti5003 returned to service after accidental reboot [production]
16:41 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2047.codfw.wmnet [production]