601-650 of 10000 results (34ms)
2022-03-10 ยง
13:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P22315 and previous config saved to /var/cache/conftool/dbconfig/20220310-133534-marostegui.json [production]
13:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P22314 and previous config saved to /var/cache/conftool/dbconfig/20220310-133254-marostegui.json [production]
13:27 <akosiaris> depool ores in codfw from discovery records to initiate reboot of rdb2007 [production]
13:26 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=ores,name=codfw [production]
13:22 <dcausse@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
13:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1121 (T302950)', diff saved to https://phabricator.wikimedia.org/P22313 and previous config saved to /var/cache/conftool/dbconfig/20220310-132234-ladsgroup.json [production]
13:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
13:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
13:20 <dcausse@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
13:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P22311 and previous config saved to /var/cache/conftool/dbconfig/20220310-132029-marostegui.json [production]
13:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114', diff saved to https://phabricator.wikimedia.org/P22310 and previous config saved to /var/cache/conftool/dbconfig/20220310-131748-marostegui.json [production]
13:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T302950)', diff saved to https://phabricator.wikimedia.org/P22309 and previous config saved to /var/cache/conftool/dbconfig/20220310-131214-ladsgroup.json [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T298294)', diff saved to https://phabricator.wikimedia.org/P22308 and previous config saved to /var/cache/conftool/dbconfig/20220310-130523-marostegui.json [production]
13:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1114 (T300775)', diff saved to https://phabricator.wikimedia.org/P22307 and previous config saved to /var/cache/conftool/dbconfig/20220310-130243-marostegui.json [production]
12:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T298294)', diff saved to https://phabricator.wikimedia.org/P22306 and previous config saved to /var/cache/conftool/dbconfig/20220310-125909-marostegui.json [production]
12:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
12:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
12:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T298294)', diff saved to https://phabricator.wikimedia.org/P22305 and previous config saved to /var/cache/conftool/dbconfig/20220310-125901-marostegui.json [production]
12:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P22304 and previous config saved to /var/cache/conftool/dbconfig/20220310-125709-ladsgroup.json [production]
12:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P22303 and previous config saved to /var/cache/conftool/dbconfig/20220310-124355-marostegui.json [production]
12:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P22302 and previous config saved to /var/cache/conftool/dbconfig/20220310-124204-ladsgroup.json [production]
12:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P22301 and previous config saved to /var/cache/conftool/dbconfig/20220310-122850-marostegui.json [production]
12:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T302950)', diff saved to https://phabricator.wikimedia.org/P22300 and previous config saved to /var/cache/conftool/dbconfig/20220310-122659-ladsgroup.json [production]
12:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1141.eqiad.wmnet with OS bullseye [production]
12:14 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Reboots [production]
12:14 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: Reboots [production]
12:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T298294)', diff saved to https://phabricator.wikimedia.org/P22299 and previous config saved to /var/cache/conftool/dbconfig/20220310-121344-marostegui.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1114 (T300775)', diff saved to https://phabricator.wikimedia.org/P22298 and previous config saved to /var/cache/conftool/dbconfig/20220310-120228-marostegui.json [production]
12:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1114.eqiad.wmnet with reason: Maintenance [production]
12:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1114.eqiad.wmnet with reason: Maintenance [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1126 (T300775)', diff saved to https://phabricator.wikimedia.org/P22297 and previous config saved to /var/cache/conftool/dbconfig/20220310-120221-marostegui.json [production]
12:00 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1141.eqiad.wmnet with reason: host reimage [production]
11:58 <marostegui> Failover m1 master [production]
11:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1141.eqiad.wmnet with reason: host reimage [production]
11:53 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Reboots [production]
11:53 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: Reboots [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P22296 and previous config saved to /var/cache/conftool/dbconfig/20220310-114715-marostegui.json [production]
11:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.reimage for host db1141.eqiad.wmnet with OS bullseye [production]
11:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T302950)', diff saved to https://phabricator.wikimedia.org/P22294 and previous config saved to /var/cache/conftool/dbconfig/20220310-113638-ladsgroup.json [production]
11:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
11:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
11:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P22293 and previous config saved to /var/cache/conftool/dbconfig/20220310-113210-marostegui.json [production]
11:29 <ebysans@deploy1002> Finished deploy [airflow-dags/analytics@b681376]: (no justification provided) (duration: 00m 07s) [production]
11:29 <ebysans@deploy1002> Started deploy [airflow-dags/analytics@b681376]: (no justification provided) [production]
11:26 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/zotero: apply [production]
11:26 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/zotero: apply [production]
11:25 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/zotero: apply [production]
11:25 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]