3201-3250 of 10000 results (117ms)
2022-11-28 §
06:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
06:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
06:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
06:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
05:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
05:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
2022-11-27 §
03:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Maint', diff saved to https://phabricator.wikimedia.org/P41257 and previous config saved to /var/cache/conftool/dbconfig/20221127-030126-ladsgroup.json [production]
02:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Maint', diff saved to https://phabricator.wikimedia.org/P41256 and previous config saved to /var/cache/conftool/dbconfig/20221127-024621-ladsgroup.json [production]
02:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Maint', diff saved to https://phabricator.wikimedia.org/P41255 and previous config saved to /var/cache/conftool/dbconfig/20221127-023116-ladsgroup.json [production]
02:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Maint', diff saved to https://phabricator.wikimedia.org/P41254 and previous config saved to /var/cache/conftool/dbconfig/20221127-021611-ladsgroup.json [production]
2022-11-26 §
21:34 <urandom> initiating Cassandra bootstrap, aqs1021-b -- T307802 [production]
09:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
09:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
09:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
09:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
02:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2105 (T323827)', diff saved to https://phabricator.wikimedia.org/P41253 and previous config saved to /var/cache/conftool/dbconfig/20221126-023900-ladsgroup.json [production]
02:38 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
02:38 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
02:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
02:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
02:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T323827)', diff saved to https://phabricator.wikimedia.org/P41252 and previous config saved to /var/cache/conftool/dbconfig/20221126-023702-ladsgroup.json [production]
02:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P41251 and previous config saved to /var/cache/conftool/dbconfig/20221126-022156-ladsgroup.json [production]
02:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P41250 and previous config saved to /var/cache/conftool/dbconfig/20221126-020649-ladsgroup.json [production]
01:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T323827)', diff saved to https://phabricator.wikimedia.org/P41249 and previous config saved to /var/cache/conftool/dbconfig/20221126-015143-ladsgroup.json [production]
01:34 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:34 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T323827)', diff saved to https://phabricator.wikimedia.org/P41248 and previous config saved to /var/cache/conftool/dbconfig/20221126-013423-ladsgroup.json [production]
01:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1197 (T323827)', diff saved to https://phabricator.wikimedia.org/P41247 and previous config saved to /var/cache/conftool/dbconfig/20221126-013225-ladsgroup.json [production]
01:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1197.eqiad.wmnet with reason: Maintenance [production]
01:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1197.eqiad.wmnet with reason: Maintenance [production]
01:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1188 (T323827)', diff saved to https://phabricator.wikimedia.org/P41246 and previous config saved to /var/cache/conftool/dbconfig/20221126-013153-ladsgroup.json [production]
01:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P41245 and previous config saved to /var/cache/conftool/dbconfig/20221126-011917-ladsgroup.json [production]
01:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P41244 and previous config saved to /var/cache/conftool/dbconfig/20221126-011647-ladsgroup.json [production]
01:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P41243 and previous config saved to /var/cache/conftool/dbconfig/20221126-010411-ladsgroup.json [production]
01:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P41242 and previous config saved to /var/cache/conftool/dbconfig/20221126-010140-ladsgroup.json [production]
00:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T323827)', diff saved to https://phabricator.wikimedia.org/P41241 and previous config saved to /var/cache/conftool/dbconfig/20221126-004904-ladsgroup.json [production]
00:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1188 (T323827)', diff saved to https://phabricator.wikimedia.org/P41240 and previous config saved to /var/cache/conftool/dbconfig/20221126-004634-ladsgroup.json [production]
00:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T323827)', diff saved to https://phabricator.wikimedia.org/P41239 and previous config saved to /var/cache/conftool/dbconfig/20221126-004437-ladsgroup.json [production]
00:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1198 (T323827)', diff saved to https://phabricator.wikimedia.org/P41238 and previous config saved to /var/cache/conftool/dbconfig/20221126-003417-ladsgroup.json [production]
00:34 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
00:34 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
00:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T323827)', diff saved to https://phabricator.wikimedia.org/P41237 and previous config saved to /var/cache/conftool/dbconfig/20221126-003356-ladsgroup.json [production]
00:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1188 (T323827)', diff saved to https://phabricator.wikimedia.org/P41236 and previous config saved to /var/cache/conftool/dbconfig/20221126-003009-ladsgroup.json [production]
00:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1188.eqiad.wmnet with reason: Maintenance [production]
00:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1188.eqiad.wmnet with reason: Maintenance [production]
00:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1182 (T323827)', diff saved to https://phabricator.wikimedia.org/P41235 and previous config saved to /var/cache/conftool/dbconfig/20221126-002948-ladsgroup.json [production]
00:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P41234 and previous config saved to /var/cache/conftool/dbconfig/20221126-002932-ladsgroup.json [production]
00:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P41233 and previous config saved to /var/cache/conftool/dbconfig/20221126-001849-ladsgroup.json [production]
00:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P41232 and previous config saved to /var/cache/conftool/dbconfig/20221126-001441-ladsgroup.json [production]
00:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P41231 and previous config saved to /var/cache/conftool/dbconfig/20221126-001425-ladsgroup.json [production]