1-50 of 10000 results (34ms)
2026-07-08 ยง
10:12 <mvernon@cumin2003> START - Cookbook sre.hosts.reimage for host thanos-be1005.eqiad.wmnet with OS trixie [production]
10:12 <cwilliams@cumin1003> END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet [production]
10:04 <cwilliams@cumin1003> START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet [production]
10:01 <cwilliams@cumin1003> END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test1001.eqiad.wmnet [production]
10:01 <atsuko@cumin1003> START - Cookbook sre.hosts.reimage for host cirrussearch1074.eqiad.wmnet with OS trixie [production]
10:01 <cwilliams@cumin1003> START - Cookbook sre.mysql.multiinstance_reboot for db-test1001.eqiad.wmnet [production]
09:43 <cgoubert@deploy1003> helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply [production]
09:43 <cgoubert@deploy1003> helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply [production]
09:43 <cgoubert@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply [production]
09:35 <cgoubert@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply [production]
09:34 <fceratto@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet [production]
09:34 <fceratto@cumin1003> START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet [production]
09:33 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 41 days, 15:00:00 on db2252.codfw.wmnet with reason: Test [production]
09:32 <fceratto@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet [production]
09:32 <fceratto@cumin1003> START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet [production]
09:31 <ayounsi@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2051: codfw rack B3 pool after maintenance [production]
09:27 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:26 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:08 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [tools]
09:07 <fceratto@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2252.codfw.wmnet [production]
09:07 <fceratto@cumin1003> START - Cookbook sre.hosts.remove-downtime for db2252.codfw.wmnet [production]
09:02 <ladsgroup@cumin1003> END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) [production]
08:57 <topranks> merge patch to shift eqiad <-> esams traffic onto new 40G circuit [production]
08:55 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [tools]
08:54 <hashar@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.10 refs T430829 [production]
08:50 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe1004.eqiad.wmnet with OS trixie [production]
08:50 <ladsgroup@cumin1003> START - Cookbook sre.mysql.sanitarium_restart [production]
08:50 <ladsgroup@cumin1003> END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) [production]
08:50 <ladsgroup@cumin1003> START - Cookbook sre.mysql.sanitarium_restart [production]
08:45 <ayounsi@cumin1003> START - Cookbook sre.mysql.pool pool es2051: codfw rack B3 pool after maintenance [production]
08:44 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet [production]
08:44 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2049,2064-2065,2262-2269].codfw.wmnet [production]
08:43 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet [production]
08:43 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet [production]
08:42 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet [production]
08:41 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet [production]
08:40 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet [production]
08:38 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [production]
08:38 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [production]
08:36 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [toolsbeta]
08:35 <fceratto@cumin1003> END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis minwikiquote in section s3 [production]
08:33 <fceratto@cumin1003> START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s3 [production]
08:32 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis minwikiquote in section s5 [production]
08:30 <fceratto@cumin1003> START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis minwikiquote in section s5 [production]
08:30 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis minwikiquote in section s5 [production]
08:27 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage [production]
08:24 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
08:23 <mvernon@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe1004.eqiad.wmnet with reason: host reimage [production]
08:23 <bwojtowicz@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
08:22 <fceratto@cumin1003> START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis minwikiquote in section s5 [production]