2901-2950 of 10000 results (141ms)
2025-01-30 ยง
12:08 <hnowlan@deploy2002> Started scap sync-world: testing removal of scap proxies [production]
11:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72859 and previous config saved to /var/cache/conftool/dbconfig/20250130-115846-root.json [production]
11:53 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72858 and previous config saved to /var/cache/conftool/dbconfig/20250130-115348-root.json [production]
11:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P72857 and previous config saved to /var/cache/conftool/dbconfig/20250130-115340-marostegui.json [production]
11:52 <elukey@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
11:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72856 and previous config saved to /var/cache/conftool/dbconfig/20250130-114341-root.json [production]
11:40 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet [production]
11:40 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet [production]
11:39 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet [production]
11:38 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72855 and previous config saved to /var/cache/conftool/dbconfig/20250130-113842-root.json [production]
11:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72854 and previous config saved to /var/cache/conftool/dbconfig/20250130-113833-marostegui.json [production]
11:35 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2224.codfw.wmnet with reason: Index rebuild [production]
11:34 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2224.codfw.wmnet [production]
11:29 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db2224.codfw.wmnet [production]
11:28 <ladsgroup@deploy2002> Synchronized portals/wikipedia.org/assets: Bump portals (second try) (duration: 11m 07s) [production]
11:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2224', diff saved to https://phabricator.wikimedia.org/P72853 and previous config saved to /var/cache/conftool/dbconfig/20250130-112853-marostegui.json [production]
11:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2191 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72852 and previous config saved to /var/cache/conftool/dbconfig/20250130-112836-root.json [production]
11:28 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2191.codfw.wmnet [production]
11:23 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72851 and previous config saved to /var/cache/conftool/dbconfig/20250130-112337-root.json [production]
11:22 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db2191.codfw.wmnet [production]
11:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2191', diff saved to https://phabricator.wikimedia.org/P72850 and previous config saved to /var/cache/conftool/dbconfig/20250130-112252-marostegui.json [production]
11:22 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Index rebuild [production]
11:21 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2172.codfw.wmnet [production]
11:16 <ladsgroup@deploy2002> Synchronized portals/wikipedia.org/assets: Bump portals (T368221 and T373204) (duration: 02m 57s) [production]
11:16 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1153.eqiad.wmnet [production]
11:13 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db2172.codfw.wmnet [production]
11:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2172', diff saved to https://phabricator.wikimedia.org/P72849 and previous config saved to /var/cache/conftool/dbconfig/20250130-111244-marostegui.json [production]
11:10 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db1153.eqiad.wmnet [production]
11:09 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2144.codfw.wmnet [production]
11:09 <root@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1151.eqiad.wmnet [production]
11:08 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72848 and previous config saved to /var/cache/conftool/dbconfig/20250130-110832-root.json [production]
11:03 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet [production]
11:02 <root@cumin1002> START - Cookbook sre.mysql.upgrade for db1151.eqiad.wmnet [production]
10:57 <elukey@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:56 <elukey@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
10:53 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72847 and previous config saved to /var/cache/conftool/dbconfig/20250130-105327-root.json [production]
10:44 <moritzm> installing util-linux bugfix updates from bookworm point release [production]
10:39 <jynus@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2201.codfw.wmnet [production]
10:39 <jynus@cumin1002> START - Cookbook sre.hosts.remove-downtime for db2201.codfw.wmnet [production]
10:38 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: prepare for decom [production]
10:37 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: test new s4 backups [production]
10:19 <cmooney@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host frbast1002 [production]
10:19 <cmooney@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host frbast1002 [production]
10:16 <cmooney@cumin1002> END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host frbast1002 [production]
10:15 <cmooney@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host frbast1002 [production]
10:04 <cmooney@cumin1002> END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host frbast1002 [production]
10:04 <cmooney@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host frbast1002 [production]
09:57 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2030.codfw.wmnet to cluster codfw and group A [production]
09:57 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72846 and previous config saved to /var/cache/conftool/dbconfig/20250130-095706-marostegui.json [production]
09:57 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance [production]