51-100 of 10000 results (26ms)
2026-06-09 ยง
09:47 <cgoubert@deploy1003> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
09:45 <btullis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:41 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:36 <Dreamy_Jazz> Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose --last-checked="20260603"` (after stopping previous scan run) [production]
09:34 <Dreamy_Jazz> Running `mwscript-k8s extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki="commonswiki" --use-jobqueue --poll-sleep=5 --verbose` (after stopping previous scan run) [production]
09:27 <btullis@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
09:26 <btullis@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
09:17 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.global-read-only (exit_code=0) [production]
09:17 <fceratto@cumin1003> MariaDB change: Setting sections s5 as read-write [production]
09:17 <fceratto@cumin1003> START - Cookbook sre.mysql.global-read-only [production]
09:14 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet [production]
09:14 <marostegui@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
09:12 <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es1042 to es4 eqiad primary T428386', diff saved to https://phabricator.wikimedia.org/P93943 and previous config saved to /var/cache/conftool/dbconfig/20260609-091215-marostegui.json [production]
09:11 <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es1043 to es4 eqiad primary T428386', diff saved to https://phabricator.wikimedia.org/P93942 and previous config saved to /var/cache/conftool/dbconfig/20260609-091147-marostegui.json [production]
09:03 <jiji@cumin1003> conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2005.codfw.wmnet [production]
08:59 <btullis@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
08:59 <btullis@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
08:57 <marostegui@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1237.eqiad.wmnet with OS trixie [production]
08:55 <jiji@cumin1003> conftool action : set/pooled=no; selector: service=docker-registry,name=registry2005.codfw.wmnet [production]
08:55 <jiji@cumin1003> conftool action : set/pooled=yes; selector: service=docker-registry,name=registry2004.codfw.wmnet [production]
08:50 <jiji@cumin1003> conftool action : set/pooled=no; selector: service=docker-registry,name=registry2004.codfw.wmnet [production]
08:22 <jiji@cumin1003> conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw [production]
08:22 <jiji@cumin1003> conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=eqiad [production]
08:08 <jiji@cumin1003> conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=eqiad [production]
08:08 <jiji@cumin1003> conftool action : set/pooled=false; selector: dnsdisc=docker-registry,name=codfw [production]
07:59 <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:59 <ayounsi@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" [production]
07:59 <ayounsi@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix typoes - ayounsi@cumin1003" [production]
07:52 <ayounsi@cumin1003> START - Cookbook sre.dns.netbox [production]
07:47 <brouberol@dns1004> END - running authdns-update [production]
07:46 <brouberol@dns1004> START - running authdns-update [production]
07:44 <brouberol@deploy1003> helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:43 <brouberol@deploy1003> helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:43 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:42 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:41 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:39 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/kafka-ui: apply [production]
07:38 <brouberol@deploy1003> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
07:37 <brouberol@deploy1003> helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [production]
07:37 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1237.eqiad.wmnet with OS trixie [production]
07:36 <marostegui@cumin1003> END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) [production]
07:36 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
07:36 <brouberol@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
07:36 <marostegui@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
07:26 <fceratto@dns1004> END - running authdns-update [production]
07:24 <fceratto@dns1004> START - running authdns-update [production]
07:22 <marostegui@dns1004> END - running authdns-update [production]
07:21 <marostegui@dns1004> START - running authdns-update [production]
07:19 <elukey@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:19 <elukey@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fix dse-k8s-wdqs2002 duplicate ipv6 address - elukey@cumin1003" [production]