51-100 of 10000 results (102ms)
2026-06-24 ยง
18:35 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm [production]
18:31 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet with OS bookworm [production]
18:25 <cwilliams@cumin1003> START - Cookbook sre.hosts.reimage for host db2219.codfw.wmnet with OS trixie [production]
18:23 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2219: Upgrading db2219.codfw.wmnet [production]
18:23 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db2219: Upgrading db2219.codfw.wmnet [production]
18:23 <cwilliams@cumin1003> dbmaint on s4@codfw T429893 [production]
18:23 <cwilliams@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
18:22 <cwilliams@cumin1003> START - Cookbook sre.hosts.reimage for host db1242.eqiad.wmnet with OS trixie [production]
18:22 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1242: Upgrading db1242.eqiad.wmnet [production]
18:22 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1016.eqiad.wmnet with reason: host reimage [production]
18:22 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db1242: Upgrading db1242.eqiad.wmnet [production]
18:21 <cwilliams@cumin1003> dbmaint on s4@eqiad T429893 [production]
18:21 <cwilliams@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
18:19 <cwilliams@cumin1003> END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) [production]
18:19 <kamila@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
18:17 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1015.eqiad.wmnet with reason: host reimage [production]
18:16 <kamila@deploy1003> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
18:14 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1017.eqiad.wmnet with reason: host reimage [production]
18:12 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.8 refs T423917 [production]
18:10 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1016.eqiad.wmnet with reason: host reimage [production]
18:09 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1015.eqiad.wmnet with reason: host reimage [production]
18:08 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1017.eqiad.wmnet with reason: host reimage [production]
18:06 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1242: Upgrading db1242.eqiad.wmnet [production]
18:05 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db1242: Upgrading db1242.eqiad.wmnet [production]
18:05 <cwilliams@cumin1003> dbmaint on s4@eqiad T429893 [production]
18:05 <cwilliams@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
18:01 <brennen> 1.47.0-wmf.8 train status (T423917): no current blockers, logs no worse than expected, rolling to group1 [production]
17:55 <cdanis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [production]
17:55 <cdanis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [production]
17:54 <cdanis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [production]
17:54 <cdanis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [production]
17:52 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1017.eqiad.wmnet with OS bookworm [production]
17:52 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1016.eqiad.wmnet with OS bookworm [production]
17:52 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm [production]
17:51 <aokoth@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T430072 [production]
17:46 <cscott@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:46 <btullis@cumin1003> END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1016.eqiad.wmnet [production]
17:46 <btullis@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet [production]
17:45 <cscott@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:45 <cscott@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:45 <cscott@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:42 <aokoth@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T430072 [production]
17:38 <kamila@deploy1003> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
17:38 <aokoth@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T430072 [production]
17:36 <kamila@deploy1003> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
17:36 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) [production]
17:36 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2210: Migration of db2210.codfw.wmnet completed [production]
17:35 <kamila@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:35 <kamila@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:34 <kamila@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]