4751-4800 of 10000 results (36ms)
2022-01-20 ยง
12:08 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply on production [production]
12:07 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: sync on staging [production]
12:06 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on production [production]
12:06 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply on staging [production]
12:06 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on staging [production]
12:05 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on production [production]
12:05 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply on staging [production]
12:05 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on staging [production]
12:05 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on production [production]
12:05 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply on staging [production]
12:04 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on staging [production]
12:04 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on production [production]
12:04 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply on staging [production]
12:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18942 and previous config saved to /var/cache/conftool/dbconfig/20220120-120015-marostegui.json [production]
11:49 <moritzm> add ganeti1024 to Ganeti eqiad cluster T283036 [production]
11:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18941 and previous config saved to /var/cache/conftool/dbconfig/20220120-114510-marostegui.json [production]
11:35 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1024.eqiad.wmnet [production]
11:30 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
11:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T285149)', diff saved to https://phabricator.wikimedia.org/P18940 and previous config saved to /var/cache/conftool/dbconfig/20220120-113006-marostegui.json [production]
11:28 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
11:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1181 (T285149)', diff saved to https://phabricator.wikimedia.org/P18939 and previous config saved to /var/cache/conftool/dbconfig/20220120-112854-marostegui.json [production]
11:28 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:28 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
11:28 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
11:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T285149)', diff saved to https://phabricator.wikimedia.org/P18938 and previous config saved to /var/cache/conftool/dbconfig/20220120-112846-marostegui.json [production]
11:28 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:24 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: sync on production [production]
11:23 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply on staging [production]
11:23 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply on production [production]
11:22 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 08s) [production]
11:22 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
11:21 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 03s) [production]
11:21 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
11:19 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: sync on production [production]
11:18 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 08s) [production]
11:18 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
11:18 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply on staging [production]
11:18 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply on production [production]
11:16 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: sync on staging [production]
11:13 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply on production [production]
11:13 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply on staging [production]
11:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18937 and previous config saved to /var/cache/conftool/dbconfig/20220120-111341-marostegui.json [production]
10:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18936 and previous config saved to /var/cache/conftool/dbconfig/20220120-105837-marostegui.json [production]
10:52 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 08s) [production]
10:52 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
10:45 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1018.eqiad.wmnet with OS buster [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T285149)', diff saved to https://phabricator.wikimedia.org/P18935 and previous config saved to /var/cache/conftool/dbconfig/20220120-104332-marostegui.json [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1158 (T285149)', diff saved to https://phabricator.wikimedia.org/P18934 and previous config saved to /var/cache/conftool/dbconfig/20220120-104220-marostegui.json [production]
10:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]