4751-4800 of 10000 results (124ms)
2024-07-30 ยง
12:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1193.eqiad.wmnet with reason: Change binlog format [production]
12:22 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db1193.eqiad.wmnet with reason: Change binlog format [production]
12:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1193 T371361', diff saved to https://phabricator.wikimedia.org/P67068 and previous config saved to /var/cache/conftool/dbconfig/20240730-122243-root.json [production]
12:21 <JustHannah> T371253 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=dewiktionary --logwiki=metawiki 'Gregorjohannes' 'Klegul' [production]
12:17 <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1057970|[Growth] hywwiki: Disable Add link backend (T370558)]], [[gerrit:1058117|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]], [[gerrit:1058116|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]] [production]
12:16 <urbanecm@deploy1003> sync-world aborted: Backport for [[gerrit:1057970|[Growth] hywwiki: Disable Add link backend (T370558)]], [[gerrit:1058117|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]], [[gerrit:1058116|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]] (duration: 14m 10s) [production]
12:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1231 T371361', diff saved to https://phabricator.wikimedia.org/P67066 and previous config saved to /var/cache/conftool/dbconfig/20240730-121500-root.json [production]
12:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P67065 and previous config saved to /var/cache/conftool/dbconfig/20240730-121201-root.json [production]
12:08 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1201.eqiad.wmnet with reason: Change binlog format [production]
12:08 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on db1201.eqiad.wmnet with reason: Change binlog format [production]
12:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1201 T371361', diff saved to https://phabricator.wikimedia.org/P67064 and previous config saved to /var/cache/conftool/dbconfig/20240730-120805-root.json [production]
12:02 <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1057970|[Growth] hywwiki: Disable Add link backend (T370558)]], [[gerrit:1058117|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]], [[gerrit:1058116|refreshLinkRecommendations: Work even when link-recommendation is disabled (T371316)]] [production]
11:54 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
11:52 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
11:47 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
11:47 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
11:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P67062 and previous config saved to /var/cache/conftool/dbconfig/20240730-111622-root.json [production]
11:13 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P67061 and previous config saved to /var/cache/conftool/dbconfig/20240730-111331-root.json [production]
11:10 <kevinbazira@deploy1003> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' . [production]
11:03 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
11:03 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
11:01 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
11:01 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:01 <kevinbazira@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' . [production]
11:01 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
11:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P67060 and previous config saved to /var/cache/conftool/dbconfig/20240730-110117-root.json [production]
11:00 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:00 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
11:00 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
10:59 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:58 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
10:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db1224 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P67059 and previous config saved to /var/cache/conftool/dbconfig/20240730-105825-root.json [production]
10:56 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:55 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:54 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:53 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:51 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:50 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2227.codfw.wmnet with OS bookworm [production]
10:50 <volans@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - volans@cumin2002" [production]