4451-4500 of 10000 results (68ms)
2022-06-16 §
09:32 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS buster [production]
09:21 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS buster [production]
09:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29868 and previous config saved to /var/cache/conftool/dbconfig/20220616-091131-marostegui.json [production]
09:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:11 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:02 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti6002.drmrs.wmnet [production]
08:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet [production]
08:45 <moritzm> failover ganeti master in drmrs/2 to ganeti6004 [production]
07:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:24 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:22 <kartik@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:805370|testwiki: Enable SectionTranslation for 11 Wikipedias (T309384 T310116)]] (duration: 03m 41s) [production]
07:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
06:49 <joal> Rerun webrequest-load-wf-upload-2022-6-15-22 after weird oozie failure [production]
2022-06-15 §
22:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T310011)', diff saved to https://phabricator.wikimedia.org/P29867 and previous config saved to /var/cache/conftool/dbconfig/20220615-224845-marostegui.json [production]
22:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P29866 and previous config saved to /var/cache/conftool/dbconfig/20220615-223339-marostegui.json [production]
22:31 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1015.eqiad.wmnet with OS buster [production]
22:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P29865 and previous config saved to /var/cache/conftool/dbconfig/20220615-221834-marostegui.json [production]
22:17 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1014.eqiad.wmnet with OS buster [production]
22:17 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1015.eqiad.wmnet with reason: host reimage [production]
22:17 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs1016.eqiad.wmnet with OS buster [production]
22:16 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host aqs1016.eqiad.wmnet with OS buster [production]
22:14 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1015.eqiad.wmnet with reason: host reimage [production]
22:12 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1016.eqiad.wmnet with OS buster [production]
22:05 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1014.eqiad.wmnet with reason: host reimage [production]
22:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T310011)', diff saved to https://phabricator.wikimedia.org/P29864 and previous config saved to /var/cache/conftool/dbconfig/20220615-220329-marostegui.json [production]
22:03 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1016.eqiad.wmnet with OS buster [production]
22:02 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1015.eqiad.wmnet with OS buster [production]
22:02 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1014.eqiad.wmnet with reason: host reimage [production]
21:49 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs1014.eqiad.wmnet with OS buster [production]
21:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1184 (T310011)', diff saved to https://phabricator.wikimedia.org/P29863 and previous config saved to /var/cache/conftool/dbconfig/20220615-213241-marostegui.json [production]
21:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
21:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
21:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T310011)', diff saved to https://phabricator.wikimedia.org/P29862 and previous config saved to /var/cache/conftool/dbconfig/20220615-213233-marostegui.json [production]
21:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P29861 and previous config saved to /var/cache/conftool/dbconfig/20220615-211728-marostegui.json [production]
21:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P29860 and previous config saved to /var/cache/conftool/dbconfig/20220615-210223-marostegui.json [production]
20:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T310011)', diff saved to https://phabricator.wikimedia.org/P29859 and previous config saved to /var/cache/conftool/dbconfig/20220615-204717-marostegui.json [production]
20:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:804014|Remove unused setting wgQuickSurveysUseVue (T285890)]] (duration: 03m 38s) [production]
20:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:50 <hashar@deploy1002> Finished deploy [integration/docroot@b95391b]: Add Developer Portal - T302809 (duration: 00m 10s) [production]
19:50 <hashar@deploy1002> Started deploy [integration/docroot@b95391b]: Add Developer Portal - T302809 [production]
19:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1132 (T310011)', diff saved to https://phabricator.wikimedia.org/P29858 and previous config saved to /var/cache/conftool/dbconfig/20220615-194703-marostegui.json [production]
19:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]