1951-2000 of 10000 results (82ms)
2023-01-11 ยง
18:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
18:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
18:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P43004 and previous config saved to /var/cache/conftool/dbconfig/20230111-184701-marostegui.json [production]
18:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 100%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P43003 and previous config saved to /var/cache/conftool/dbconfig/20230111-184051-root.json [production]
18:36 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@5a19b9d]: drop-snapshots: Accept snapshot= partition from any level (duration: 02m 33s) [production]
18:33 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@5a19b9d]: drop-snapshots: Accept snapshot= partition from any level [production]
18:33 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
18:32 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
18:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P43002 and previous config saved to /var/cache/conftool/dbconfig/20230111-183155-marostegui.json [production]
18:30 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
18:30 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
18:28 <bblack> repool eqsin edge DC [production]
18:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 75%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P43001 and previous config saved to /var/cache/conftool/dbconfig/20230111-182546-root.json [production]
18:22 <btullis@cumin1001> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) [production]
18:22 <btullis@cumin1001> Added views for new wiki: blkwiki T310872 [production]
18:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P43000 and previous config saved to /var/cache/conftool/dbconfig/20230111-181648-marostegui.json [production]
18:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 50%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42999 and previous config saved to /var/cache/conftool/dbconfig/20230111-181041-root.json [production]
18:09 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: sync [production]
18:09 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
18:08 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
18:08 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
18:07 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
18:02 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
18:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P42998 and previous config saved to /var/cache/conftool/dbconfig/20230111-180142-marostegui.json [production]
18:01 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
17:59 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: sync [production]
17:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P42997 and previous config saved to /var/cache/conftool/dbconfig/20230111-175919-marostegui.json [production]
17:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
17:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
17:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2112 (T321391)', diff saved to https://phabricator.wikimedia.org/P42996 and previous config saved to /var/cache/conftool/dbconfig/20230111-175857-marostegui.json [production]
17:58 <btullis@cumin1001> START - Cookbook sre.wikireplicas.add-wiki [production]
17:56 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement [production]
17:55 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement [production]
17:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 25%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42995 and previous config saved to /var/cache/conftool/dbconfig/20230111-175536-root.json [production]
17:50 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
17:50 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
17:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P42994 and previous config saved to /var/cache/conftool/dbconfig/20230111-174351-marostegui.json [production]
17:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 10%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42993 and previous config saved to /var/cache/conftool/dbconfig/20230111-174031-root.json [production]
17:40 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
17:39 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
17:29 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
17:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P42992 and previous config saved to /var/cache/conftool/dbconfig/20230111-172844-marostegui.json [production]
17:28 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:28 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1106 (re)pooling @ 5%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42991 and previous config saved to /var/cache/conftool/dbconfig/20230111-172526-root.json [production]
17:21 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:21 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:21 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:20 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:18 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]