1351-1400 of 10000 results (136ms)
2026-06-02 ยง
12:07 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011,2033-2034,2050,2055-2062,2068-2071,2107-2113].codfw.wmnet [production]
12:07 <dreamyjazz@deploy1003> Started scap sync-world: Backport for [[gerrit:1296532|hCaptcha: Deduplicate edit API detection code (T427887)]], [[gerrit:1296533|hCaptcha: Disable hCaptcha for DiscussionTools for the apps (T427887)]] [production]
12:05 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on es2049.codfw.wmnet with reason: host reimage [production]
12:04 <blake@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage [production]
12:02 <blake@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage [production]
12:01 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS trixie [production]
12:00 <blake@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage [production]
11:58 <trueg@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
11:57 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93535 and previous config saved to /var/cache/conftool/dbconfig/20260602-115721-fceratto.json [production]
11:55 <jelto@deploy1003> helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply [production]
11:55 <trueg@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
11:55 <trueg@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
11:55 <jelto@deploy1003> helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply [production]
11:53 <jelto@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply [production]
11:53 <jelto@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply [production]
11:53 <trueg@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
11:50 <blake@cumin1003> START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS trixie [production]
11:49 <blake@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS trixie [production]
11:49 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host es2049.codfw.wmnet with OS trixie [production]
11:48 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2049: Upgrading es2049.codfw.wmnet [production]
11:48 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool es2049: Upgrading es2049.codfw.wmnet [production]
11:47 <marostegui@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
11:47 <blake@cumin1003> START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS trixie [production]
11:47 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2056: repool after upgrade [production]
11:47 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P93532 and previous config saved to /var/cache/conftool/dbconfig/20260602-114713-fceratto.json [production]
11:45 <blake@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS trixie [production]
11:44 <cwilliams@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage [production]
11:40 <cwilliams@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage [production]
11:37 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T426633)', diff saved to https://phabricator.wikimedia.org/P93531 and previous config saved to /var/cache/conftool/dbconfig/20260602-113705-fceratto.json [production]
11:33 <blake@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage [production]
11:30 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1185 (T426633)', diff saved to https://phabricator.wikimedia.org/P93529 and previous config saved to /var/cache/conftool/dbconfig/20260602-113019-fceratto.json [production]
11:30 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
11:29 <blake@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage [production]
11:26 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Repooling [production]
11:26 <fceratto@cumin1003> START - Cookbook sre.mysql.pool pool db1161: Repooling [production]
11:23 <cwilliams@cumin1003> START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS trixie [production]
11:22 <blake@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage [production]
11:21 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2161: Upgrading db2161.codfw.wmnet [production]
11:21 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db2161: Upgrading db2161.codfw.wmnet [production]
11:21 <blake@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage [production]
11:21 <cwilliams@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
11:19 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93527 and previous config saved to /var/cache/conftool/dbconfig/20260602-111954-fceratto.json [production]
11:15 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depool db2161 T427892', diff saved to https://phabricator.wikimedia.org/P93525 and previous config saved to /var/cache/conftool/dbconfig/20260602-111511-cwilliams.json [production]
11:12 <cwilliams@cumin1003> dbctl commit (dc=all): 'Promote db2165 to s8 primary T427892', diff saved to https://phabricator.wikimedia.org/P93524 and previous config saved to /var/cache/conftool/dbconfig/20260602-111200-cwilliams.json [production]
11:10 <cezmunsta> Starting s8 codfw failover from db2161 to db2165 - T427892 [production]
11:09 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P93523 and previous config saved to /var/cache/conftool/dbconfig/20260602-110947-fceratto.json [production]
11:09 <blake@cumin1003> START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS trixie [production]
11:09 <blake@cumin1003> START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS trixie [production]
11:04 <cwilliams@cumin1003> dbctl commit (dc=all): 'Set db2165 with weight 0 T427892', diff saved to https://phabricator.wikimedia.org/P93522 and previous config saved to /var/cache/conftool/dbconfig/20260602-110420-cwilliams.json [production]
11:03 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s8 T427892 [production]