8151-8200 of 10000 results (105ms)
2022-11-28 ยง
08:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
08:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321126)', diff saved to https://phabricator.wikimedia.org/P41272 and previous config saved to /var/cache/conftool/dbconfig/20221128-084616-marostegui.json [production]
08:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet [production]
08:39 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
08:35 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet [production]
08:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
08:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
08:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P41271 and previous config saved to /var/cache/conftool/dbconfig/20221128-083110-marostegui.json [production]
08:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
08:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
08:25 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
08:24 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
08:22 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
08:22 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
08:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
08:21 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
08:21 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
08:21 <kartik@deploy1002> Finished scap: Backport for [[gerrit:861341|Revert "Content Translation: Reverse MT threshold for Japanese Wikipedia"]] (duration: 11m 12s) [production]
08:21 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
08:19 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply [production]
08:19 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/recommendation-api: apply [production]
08:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
08:16 <kartik@deploy1002> kartik and trainbranchbot: Backport for [[gerrit:861341|Revert "Content Translation: Reverse MT threshold for Japanese Wikipedia"]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P41270 and previous config saved to /var/cache/conftool/dbconfig/20221128-081603-marostegui.json [production]
08:12 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
08:12 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply [production]
08:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
08:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
08:11 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/recommendation-api: apply [production]
08:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
08:10 <kartik@deploy1002> Started scap: Backport for [[gerrit:861341|Revert "Content Translation: Reverse MT threshold for Japanese Wikipedia"]] [production]
08:09 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
08:09 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
08:07 <kartik@deploy1002> Backport cancelled. [production]
08:04 <moritzm> rebalance Ganeti group C/codfw following reboots [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321126)', diff saved to https://phabricator.wikimedia.org/P41269 and previous config saved to /var/cache/conftool/dbconfig/20221128-080057-marostegui.json [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1142 (T321126)', diff saved to https://phabricator.wikimedia.org/P41268 and previous config saved to /var/cache/conftool/dbconfig/20221128-075847-marostegui.json [production]
07:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
07:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41267 and previous config saved to /var/cache/conftool/dbconfig/20221128-075826-marostegui.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P41266 and previous config saved to /var/cache/conftool/dbconfig/20221128-074319-marostegui.json [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P41265 and previous config saved to /var/cache/conftool/dbconfig/20221128-072813-marostegui.json [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41264 and previous config saved to /var/cache/conftool/dbconfig/20221128-071306-marostegui.json [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41263 and previous config saved to /var/cache/conftool/dbconfig/20221128-071057-marostegui.json [production]
07:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
07:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321126)', diff saved to https://phabricator.wikimedia.org/P41262 and previous config saved to /var/cache/conftool/dbconfig/20221128-071035-marostegui.json [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P41261 and previous config saved to /var/cache/conftool/dbconfig/20221128-065529-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P41260 and previous config saved to /var/cache/conftool/dbconfig/20221128-064022-marostegui.json [production]
06:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321126)', diff saved to https://phabricator.wikimedia.org/P41259 and previous config saved to /var/cache/conftool/dbconfig/20221128-062516-marostegui.json [production]