251-300 of 10000 results (71ms)
2024-04-10 ยง
12:01 <claime> Running homer 'cr*eqiad*' commit 'T351074' and homer 'lsw1-e3-eqiad*' commit 'T351074' [production]
11:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60225 and previous config saved to /var/cache/conftool/dbconfig/20240410-115506-root.json [production]
11:54 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS bullseye [production]
11:53 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1491.eqiad.wmnet with OS bullseye [production]
11:49 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1422.eqiad.wmnet with OS bullseye [production]
11:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60224 and previous config saved to /var/cache/conftool/dbconfig/20240410-114728-arnaudb.json [production]
11:45 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1493.eqiad.wmnet with OS bullseye [production]
11:42 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1421.eqiad.wmnet with OS bullseye [production]
11:40 <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60223 and previous config saved to /var/cache/conftool/dbconfig/20240410-114001-root.json [production]
11:38 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage [production]
11:34 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage [production]
11:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60222 and previous config saved to /var/cache/conftool/dbconfig/20240410-113220-arnaudb.json [production]
11:31 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage [production]
11:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60221 and previous config saved to /var/cache/conftool/dbconfig/20240410-112929-arnaudb.json [production]
11:29 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
11:29 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
11:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60220 and previous config saved to /var/cache/conftool/dbconfig/20240410-112907-arnaudb.json [production]
11:28 <jiji@deploy1002> Finished scap: Deploy chart changes in gerrit:1015342 (duration: 08m 18s) [production]
11:27 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage [production]
11:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60219 and previous config saved to /var/cache/conftool/dbconfig/20240410-112455-root.json [production]
11:24 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage [production]
11:23 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage [production]
11:22 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage [production]
11:22 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage [production]
11:21 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage [production]
11:21 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage [production]
11:19 <jiji@deploy1002> Started scap: Deploy chart changes in gerrit:1015342 [production]
11:16 <mvolz@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
11:15 <mvolz@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
11:14 <mvolz@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
11:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60218 and previous config saved to /var/cache/conftool/dbconfig/20240410-111400-arnaudb.json [production]
11:13 <mvolz@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
11:12 <mvolz@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
11:12 <mvolz@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
11:10 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host mw1493.eqiad.wmnet with OS bullseye [production]
11:09 <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60217 and previous config saved to /var/cache/conftool/dbconfig/20240410-110949-root.json [production]
11:09 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS bullseye [production]
11:09 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host mw1491.eqiad.wmnet with OS bullseye [production]
11:08 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:08 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host mw1422.eqiad.wmnet with OS bullseye [production]
11:08 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host mw1421.eqiad.wmnet with OS bullseye [production]
11:07 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
11:07 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:07 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
11:03 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:02 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
11:02 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:02 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
10:59 <claime> Depooling mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074 [production]
10:58 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60216 and previous config saved to /var/cache/conftool/dbconfig/20240410-105852-arnaudb.json [production]