8201-8250 of 10000 results (76ms)
2022-06-27 ยง
12:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30424 and previous config saved to /var/cache/conftool/dbconfig/20220627-124723-root.json [production]
12:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30423 and previous config saved to /var/cache/conftool/dbconfig/20220627-124719-root.json [production]
12:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30422 and previous config saved to /var/cache/conftool/dbconfig/20220627-124715-root.json [production]
12:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30421 and previous config saved to /var/cache/conftool/dbconfig/20220627-124648-ladsgroup.json [production]
12:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30420 and previous config saved to /var/cache/conftool/dbconfig/20220627-124132-ladsgroup.json [production]
12:41 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
12:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
12:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30419 and previous config saved to /var/cache/conftool/dbconfig/20220627-124124-ladsgroup.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30418 and previous config saved to /var/cache/conftool/dbconfig/20220627-123230-root.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30417 and previous config saved to /var/cache/conftool/dbconfig/20220627-123224-root.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30416 and previous config saved to /var/cache/conftool/dbconfig/20220627-123219-root.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30415 and previous config saved to /var/cache/conftool/dbconfig/20220627-123215-root.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30414 and previous config saved to /var/cache/conftool/dbconfig/20220627-123211-root.json [production]
12:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P30413 and previous config saved to /var/cache/conftool/dbconfig/20220627-122618-ladsgroup.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30412 and previous config saved to /var/cache/conftool/dbconfig/20220627-121726-root.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30411 and previous config saved to /var/cache/conftool/dbconfig/20220627-121720-root.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30410 and previous config saved to /var/cache/conftool/dbconfig/20220627-121715-root.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30409 and previous config saved to /var/cache/conftool/dbconfig/20220627-121711-root.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30408 and previous config saved to /var/cache/conftool/dbconfig/20220627-121708-root.json [production]
12:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P30407 and previous config saved to /var/cache/conftool/dbconfig/20220627-121109-ladsgroup.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1167 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30406 and previous config saved to /var/cache/conftool/dbconfig/20220627-120222-root.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30405 and previous config saved to /var/cache/conftool/dbconfig/20220627-120216-root.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30404 and previous config saved to /var/cache/conftool/dbconfig/20220627-120211-root.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30403 and previous config saved to /var/cache/conftool/dbconfig/20220627-120207-root.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30402 and previous config saved to /var/cache/conftool/dbconfig/20220627-120201-root.json [production]
11:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30401 and previous config saved to /var/cache/conftool/dbconfig/20220627-115604-ladsgroup.json [production]
11:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30400 and previous config saved to /var/cache/conftool/dbconfig/20220627-115148-ladsgroup.json [production]
11:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
11:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
11:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30399 and previous config saved to /var/cache/conftool/dbconfig/20220627-115140-ladsgroup.json [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1167 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30398 and previous config saved to /var/cache/conftool/dbconfig/20220627-114718-root.json [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30397 and previous config saved to /var/cache/conftool/dbconfig/20220627-114712-root.json [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30396 and previous config saved to /var/cache/conftool/dbconfig/20220627-114707-root.json [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30395 and previous config saved to /var/cache/conftool/dbconfig/20220627-114703-root.json [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30394 and previous config saved to /var/cache/conftool/dbconfig/20220627-114658-root.json [production]
11:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134 db1147 db1158 d1165 db1167 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30393 and previous config saved to /var/cache/conftool/dbconfig/20220627-113834-root.json [production]
11:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P30392 and previous config saved to /var/cache/conftool/dbconfig/20220627-113634-ladsgroup.json [production]
11:31 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/image-suggestion: sync [production]
11:30 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/image-suggestion: sync [production]
11:29 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/image-suggestion: sync [production]
11:29 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/image-suggestion: sync [production]
11:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P30391 and previous config saved to /var/cache/conftool/dbconfig/20220627-112129-ladsgroup.json [production]
11:14 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/image-suggestion: sync [production]
11:13 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/image-suggestion: sync [production]
11:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30390 and previous config saved to /var/cache/conftool/dbconfig/20220627-110624-ladsgroup.json [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30389 and previous config saved to /var/cache/conftool/dbconfig/20220627-110237-root.json [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30388 and previous config saved to /var/cache/conftool/dbconfig/20220627-110232-root.json [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30387 and previous config saved to /var/cache/conftool/dbconfig/20220627-110226-root.json [production]
11:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T298565)', diff saved to https://phabricator.wikimedia.org/P30386 and previous config saved to /var/cache/conftool/dbconfig/20220627-110207-ladsgroup.json [production]
11:02 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]