1-50 of 10000 results (88ms)
2025-10-10 §
12:06 <marostegui@cumin1003> dbctl commit (dc=all): 'db1243 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P83750 and previous config saved to /var/cache/conftool/dbconfig/20251010-120643-root.json [production]
11:51 <marostegui@cumin1003> dbctl commit (dc=all): 'db1243 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P83749 and previous config saved to /var/cache/conftool/dbconfig/20251010-115138-root.json [production]
11:36 <marostegui@cumin1003> dbctl commit (dc=all): 'db1243 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P83748 and previous config saved to /var/cache/conftool/dbconfig/20251010-113632-root.json [production]
11:21 <marostegui@cumin1003> dbctl commit (dc=all): 'db1243 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P83747 and previous config saved to /var/cache/conftool/dbconfig/20251010-112126-root.json [production]
11:16 <marostegui@cumin1003> dbctl commit (dc=all): 'Change es2 eqiad master to es1030 T406488', diff saved to https://phabricator.wikimedia.org/P83746 and previous config saved to /var/cache/conftool/dbconfig/20251010-111653-marostegui.json [production]
11:16 <marostegui@cumin1003> dbctl commit (dc=all): 'Change es1 eqiad master to es1029 T406488', diff saved to https://phabricator.wikimedia.org/P83745 and previous config saved to /var/cache/conftool/dbconfig/20251010-111630-marostegui.json [production]
11:16 <klausman@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
11:16 <marostegui@cumin1003> dbctl commit (dc=all): 'Change es3 eqiad master to es1028 T406488', diff saved to https://phabricator.wikimedia.org/P83744 and previous config saved to /var/cache/conftool/dbconfig/20251010-111605-marostegui.json [production]
11:15 <klausman@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
11:15 <klausman@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:14 <klausman@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
11:13 <klausman@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:13 <klausman@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
11:13 <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db1243 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P83743 and previous config saved to /var/cache/conftool/dbconfig/20251010-111306-marostegui.json [production]
11:13 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1243.eqiad.wmnet with reason: Maintenance [production]
11:10 <marostegui@cumin1003> dbctl commit (dc=all): 'db1248 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P83742 and previous config saved to /var/cache/conftool/dbconfig/20251010-111020-root.json [production]
10:55 <marostegui@cumin1003> dbctl commit (dc=all): 'db1248 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P83741 and previous config saved to /var/cache/conftool/dbconfig/20251010-105514-root.json [production]
10:40 <marostegui@cumin1003> dbctl commit (dc=all): 'db1248 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P83740 and previous config saved to /var/cache/conftool/dbconfig/20251010-104008-root.json [production]
10:33 <fceratto@deploy2002> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
10:32 <vgutierrez> restarting acme-chief and nginx on acme-chief instances [production]
10:25 <marostegui@cumin1003> dbctl commit (dc=all): 'db1248 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P83739 and previous config saved to /var/cache/conftool/dbconfig/20251010-102502-root.json [production]
10:17 <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db1248 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P83738 and previous config saved to /var/cache/conftool/dbconfig/20251010-101720-marostegui.json [production]
10:17 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1248.eqiad.wmnet with reason: Maintenance [production]
09:34 <btullis@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
09:34 <btullis@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
09:20 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:19 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
06:24 <marostegui@cumin1003> dbctl commit (dc=all): 'db1249 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P83737 and previous config saved to /var/cache/conftool/dbconfig/20251010-062406-root.json [production]
06:11 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.clone_es (exit_code=0) of es1029.eqiad.wmnet onto es1052.eqiad.wmnet [production]
06:10 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1029 gradually with 4 steps - Pool es1029.eqiad.wmnet in after cloning [production]
06:10 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.clone_es (exit_code=0) of es1034.eqiad.wmnet onto es1057.eqiad.wmnet [production]
06:10 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1034 gradually with 4 steps - Pool es1034.eqiad.wmnet in after cloning [production]
06:09 <marostegui@cumin1003> dbctl commit (dc=all): 'db1249 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P83734 and previous config saved to /var/cache/conftool/dbconfig/20251010-060900-root.json [production]
05:53 <marostegui@cumin1003> dbctl commit (dc=all): 'db1249 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P83731 and previous config saved to /var/cache/conftool/dbconfig/20251010-055354-root.json [production]
05:38 <marostegui@cumin1003> dbctl commit (dc=all): 'db1249 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P83728 and previous config saved to /var/cache/conftool/dbconfig/20251010-053848-root.json [production]
05:30 <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db1249 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P83727 and previous config saved to /var/cache/conftool/dbconfig/20251010-053040-marostegui.json [production]
05:30 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1249.eqiad.wmnet with reason: Maintenance [production]
05:25 <marostegui@cumin1003> START - Cookbook sre.mysql.pool es1029 gradually with 4 steps - Pool es1029.eqiad.wmnet in after cloning [production]
05:25 <marostegui@cumin1003> START - Cookbook sre.mysql.pool es1034 gradually with 4 steps - Pool es1034.eqiad.wmnet in after cloning [production]
01:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 13m 32s) [production]
01:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
2025-10-09 §
23:10 <ryankemper@cumin2002> conftool action : set/pooled=yes:weight=10; selector: name=wdqs2017.* [production]
22:11 <inflatador> bking@wdqs10(18|19|20) systemctl start load-categories-daily.service T405978 [production]
22:05 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1019.eqiad.wmnet [production]
22:04 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1020.eqiad.wmnet [production]
22:04 <jdlrobson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1195049|Enable instrumentation of watchstar and other links that stopPropagation (T406390)]] (duration: 41m 38s) [production]
22:00 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1018.eqiad.wmnet [production]
21:51 <dwisehaupt> started staging db restore in root screen session on frdb1006. restoring from db backups on 20251008 [production]
21:51 <jdlrobson@deploy2002> jdlrobson: Continuing with sync [production]
21:47 <jdlrobson@deploy2002> jdlrobson: Backport for [[gerrit:1195049|Enable instrumentation of watchstar and other links that stopPropagation (T406390)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]