1201-1250 of 10000 results (100ms)
2025-02-05 ยง
12:53 <marostegui@cumin1002> dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73227 and previous config saved to /var/cache/conftool/dbconfig/20250205-125259-root.json [production]
12:50 <marostegui@cumin1002> START - Cookbook sre.mysql.clone of db1237.eqiad.wmnet onto db1179.eqiad.wmnet [production]
12:46 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1237.eqiad.wmnet onto db1179.eqiad.wmnet [production]
12:38 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 35%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73226 and previous config saved to /var/cache/conftool/dbconfig/20250205-123803-fceratto.json [production]
12:22 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 30%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73225 and previous config saved to /var/cache/conftool/dbconfig/20250205-122257-fceratto.json [production]
12:12 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
12:12 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
12:12 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
12:12 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
12:09 <mvolz@deploy2002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
12:09 <mvolz@deploy2002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
12:07 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 25%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73224 and previous config saved to /var/cache/conftool/dbconfig/20250205-120752-fceratto.json [production]
12:06 <mvolz@deploy2002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
12:05 <mvolz@deploy2002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
12:03 <mvolz@deploy2002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
12:03 <mvolz@deploy2002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
12:00 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s1 [production]
12:00 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s3 [production]
12:00 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
12:00 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
11:56 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1017.eqiad.wmnet with reason: Rebuild tables [production]
11:52 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 20%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73223 and previous config saved to /var/cache/conftool/dbconfig/20250205-115247-fceratto.json [production]
11:42 <jiji@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [production]
11:42 <jiji@deploy2002> helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [production]
11:41 <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1017.eqiad.wmnet [production]
11:38 <fnegri@cumin1002> START - Cookbook sre.hosts.reboot-single for host clouddb1017.eqiad.wmnet [production]
11:37 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 15%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73222 and previous config saved to /var/cache/conftool/dbconfig/20250205-113741-fceratto.json [production]
11:34 <jiji@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [production]
11:34 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb1014.eqiad.wmnet with reason: Rebuild tables [production]
11:34 <jiji@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [production]
11:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on clouddb1018.eqiad.wmnet with reason: Rebuild tables [production]
11:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1014.eqiad.wmnet with reason: Rebuild tables [production]
11:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Rebuild tables [production]
11:32 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1155.eqiad.wmnet with reason: Rebuild tables [production]
11:31 <fnegri@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1017.eqiad.wmnet with reason: Rebooting clouddb1017 T384946 [production]
11:31 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s3 [production]
11:31 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=31 [production]
11:31 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1 [production]
11:28 <jiji@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
11:28 <jiji@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
11:27 <jiji@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
11:27 <jiji@deploy2002> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
11:25 <jiji@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox: apply [production]
11:25 <jiji@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox: apply [production]
11:22 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 10%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73221 and previous config saved to /var/cache/conftool/dbconfig/20250205-112236-fceratto.json [production]
11:11 <godog> bounce thanos-query on titan1002 [production]
11:07 <fceratto@cumin1002> dbctl commit (dc=all): 'db1251 (re)pooling @ 7%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73220 and previous config saved to /var/cache/conftool/dbconfig/20250205-110731-fceratto.json [production]
11:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P73219 and previous config saved to /var/cache/conftool/dbconfig/20250205-110628-marostegui.json [production]
11:03 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Index rebuild [production]
11:03 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2221.codfw.wmnet with reason: Index rebuild [production]