651-700 of 10000 results (49ms)
2022-05-02 ยง
05:20 <Amir1> killed bnwiki's refresh links recommendation (T299021) [production]
05:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P27259 and previous config saved to /var/cache/conftool/dbconfig/20220502-051844-ladsgroup.json [production]
05:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P27258 and previous config saved to /var/cache/conftool/dbconfig/20220502-051402-ladsgroup.json [production]
05:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
05:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
05:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P27257 and previous config saved to /var/cache/conftool/dbconfig/20220502-051255-ladsgroup.json [production]
05:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P27256 and previous config saved to /var/cache/conftool/dbconfig/20220502-050339-ladsgroup.json [production]
04:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298295)', diff saved to https://phabricator.wikimedia.org/P27255 and previous config saved to /var/cache/conftool/dbconfig/20220502-045750-ladsgroup.json [production]
04:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P27254 and previous config saved to /var/cache/conftool/dbconfig/20220502-045656-ladsgroup.json [production]
04:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
04:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
04:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1110 (T298295)', diff saved to https://phabricator.wikimedia.org/P27253 and previous config saved to /var/cache/conftool/dbconfig/20220502-045532-ladsgroup.json [production]
04:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance [production]
04:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance [production]
04:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
04:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
04:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
04:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P27252 and previous config saved to /var/cache/conftool/dbconfig/20220502-044834-ladsgroup.json [production]
04:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P27251 and previous config saved to /var/cache/conftool/dbconfig/20220502-044614-ladsgroup.json [production]
04:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
04:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
04:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P27250 and previous config saved to /var/cache/conftool/dbconfig/20220502-044606-ladsgroup.json [production]
04:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27249 and previous config saved to /var/cache/conftool/dbconfig/20220502-044151-ladsgroup.json [production]
04:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P27248 and previous config saved to /var/cache/conftool/dbconfig/20220502-043101-ladsgroup.json [production]
04:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27247 and previous config saved to /var/cache/conftool/dbconfig/20220502-042646-ladsgroup.json [production]
04:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P27246 and previous config saved to /var/cache/conftool/dbconfig/20220502-041556-ladsgroup.json [production]
04:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298565)', diff saved to https://phabricator.wikimedia.org/P27245 and previous config saved to /var/cache/conftool/dbconfig/20220502-041141-ladsgroup.json [production]
04:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance [production]
04:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance [production]
04:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
04:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
04:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1130 (T298295)', diff saved to https://phabricator.wikimedia.org/P27244 and previous config saved to /var/cache/conftool/dbconfig/20220502-040908-ladsgroup.json [production]
04:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1130 (T298295)', diff saved to https://phabricator.wikimedia.org/P27243 and previous config saved to /var/cache/conftool/dbconfig/20220502-040754-ladsgroup.json [production]
04:07 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
04:07 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
04:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T298295)', diff saved to https://phabricator.wikimedia.org/P27242 and previous config saved to /var/cache/conftool/dbconfig/20220502-040745-ladsgroup.json [production]
04:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
04:01 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
04:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
04:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P27241 and previous config saved to /var/cache/conftool/dbconfig/20220502-040051-ladsgroup.json [production]
03:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3311 (T306560)', diff saved to https://phabricator.wikimedia.org/P27240 and previous config saved to /var/cache/conftool/dbconfig/20220502-035830-ladsgroup.json [production]
03:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
03:58 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
03:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]
03:58 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]