2301-2350 of 10000 results (101ms)
2024-07-30 ยง
11:01 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
11:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P67060 and previous config saved to /var/cache/conftool/dbconfig/20240730-110117-root.json [production]
11:00 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:00 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
11:00 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
10:59 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:58 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
10:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P67059 and previous config saved to /var/cache/conftool/dbconfig/20240730-105825-root.json [production]
10:56 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:53 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:51 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:50 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2227.codfw.wmnet with OS bookworm [production]
10:50 <volans@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - volans@cumin2002" [production]
10:49 <volans@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - volans@cumin2002" [production]
10:47 <marostegui@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P67058 and previous config saved to /var/cache/conftool/dbconfig/20240730-104705-root.json [production]
10:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P67057 and previous config saved to /var/cache/conftool/dbconfig/20240730-104612-root.json [production]
10:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P67056 and previous config saved to /var/cache/conftool/dbconfig/20240730-104318-root.json [production]
10:33 <kevinbazira@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . [production]
10:32 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2227.codfw.wmnet with reason: host reimage [production]
10:32 <marostegui@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P67054 and previous config saved to /var/cache/conftool/dbconfig/20240730-103200-root.json [production]
10:31 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P67053 and previous config saved to /var/cache/conftool/dbconfig/20240730-103106-root.json [production]
10:29 <volans@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2227.codfw.wmnet with reason: host reimage [production]
10:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P67052 and previous config saved to /var/cache/conftool/dbconfig/20240730-102813-root.json [production]
10:21 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:20 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
10:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P67051 and previous config saved to /var/cache/conftool/dbconfig/20240730-101654-root.json [production]
10:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P67050 and previous config saved to /var/cache/conftool/dbconfig/20240730-101600-root.json [production]
10:14 <volans@cumin2002> START - Cookbook sre.hosts.reimage for host db2227.codfw.wmnet with OS bookworm [production]
10:13 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P67049 and previous config saved to /var/cache/conftool/dbconfig/20240730-101307-root.json [production]
10:08 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1240.mgmt.eqiad.wmnet with reboot policy FORCED [production]
10:02 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host wikikube-worker1240.mgmt.eqiad.wmnet with reboot policy FORCED [production]
10:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P67048 and previous config saved to /var/cache/conftool/dbconfig/20240730-100148-root.json [production]
10:00 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P67047 and previous config saved to /var/cache/conftool/dbconfig/20240730-100055-root.json [production]
09:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P67046 and previous config saved to /var/cache/conftool/dbconfig/20240730-095802-root.json [production]
09:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P67045 and previous config saved to /var/cache/conftool/dbconfig/20240730-094643-root.json [production]
09:45 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P67044 and previous config saved to /var/cache/conftool/dbconfig/20240730-094549-root.json [production]