5401-5450 of 10000 results (90ms)
2022-11-28 §
08:19 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply [production]
08:19 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/recommendation-api: apply [production]
08:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
08:16 <kartik@deploy1002> kartik and trainbranchbot: Backport for [[gerrit:861341|Revert "Content Translation: Reverse MT threshold for Japanese Wikipedia"]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P41270 and previous config saved to /var/cache/conftool/dbconfig/20221128-081603-marostegui.json [production]
08:12 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
08:12 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply [production]
08:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
08:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
08:11 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/recommendation-api: apply [production]
08:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
08:10 <kartik@deploy1002> Started scap: Backport for [[gerrit:861341|Revert "Content Translation: Reverse MT threshold for Japanese Wikipedia"]] [production]
08:09 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
08:09 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
08:07 <kartik@deploy1002> Backport cancelled. [production]
08:04 <moritzm> rebalance Ganeti group C/codfw following reboots [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321126)', diff saved to https://phabricator.wikimedia.org/P41269 and previous config saved to /var/cache/conftool/dbconfig/20221128-080057-marostegui.json [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1142 (T321126)', diff saved to https://phabricator.wikimedia.org/P41268 and previous config saved to /var/cache/conftool/dbconfig/20221128-075847-marostegui.json [production]
07:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
07:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41267 and previous config saved to /var/cache/conftool/dbconfig/20221128-075826-marostegui.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P41266 and previous config saved to /var/cache/conftool/dbconfig/20221128-074319-marostegui.json [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P41265 and previous config saved to /var/cache/conftool/dbconfig/20221128-072813-marostegui.json [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41264 and previous config saved to /var/cache/conftool/dbconfig/20221128-071306-marostegui.json [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T321126)', diff saved to https://phabricator.wikimedia.org/P41263 and previous config saved to /var/cache/conftool/dbconfig/20221128-071057-marostegui.json [production]
07:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
07:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321126)', diff saved to https://phabricator.wikimedia.org/P41262 and previous config saved to /var/cache/conftool/dbconfig/20221128-071035-marostegui.json [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P41261 and previous config saved to /var/cache/conftool/dbconfig/20221128-065529-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P41260 and previous config saved to /var/cache/conftool/dbconfig/20221128-064022-marostegui.json [production]
06:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321126)', diff saved to https://phabricator.wikimedia.org/P41259 and previous config saved to /var/cache/conftool/dbconfig/20221128-062516-marostegui.json [production]
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1121 (T321126)', diff saved to https://phabricator.wikimedia.org/P41258 and previous config saved to /var/cache/conftool/dbconfig/20221128-062008-marostegui.json [production]
06:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
06:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
06:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
06:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
06:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
06:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
05:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
05:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
2022-11-27 §
03:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Maint', diff saved to https://phabricator.wikimedia.org/P41257 and previous config saved to /var/cache/conftool/dbconfig/20221127-030126-ladsgroup.json [production]
02:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Maint', diff saved to https://phabricator.wikimedia.org/P41256 and previous config saved to /var/cache/conftool/dbconfig/20221127-024621-ladsgroup.json [production]
02:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Maint', diff saved to https://phabricator.wikimedia.org/P41255 and previous config saved to /var/cache/conftool/dbconfig/20221127-023116-ladsgroup.json [production]
02:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Maint', diff saved to https://phabricator.wikimedia.org/P41254 and previous config saved to /var/cache/conftool/dbconfig/20221127-021611-ladsgroup.json [production]
2022-11-26 §
21:34 <urandom> initiating Cassandra bootstrap, aqs1021-b -- T307802 [production]
09:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
09:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
09:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
09:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
02:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2105 (T323827)', diff saved to https://phabricator.wikimedia.org/P41253 and previous config saved to /var/cache/conftool/dbconfig/20221126-023900-ladsgroup.json [production]