151-200 of 10000 results (132ms)
2026-04-16 §
16:27 <urandom> upgrade envoyproxy, restbase[1031,2024] (canary) — T419637 & T410975 [production]
16:27 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P90956 and previous config saved to /var/cache/conftool/dbconfig/20260416-162727-fceratto.json [production]
16:22 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2175 (T419961)', diff saved to https://phabricator.wikimedia.org/P90955 and previous config saved to /var/cache/conftool/dbconfig/20260416-162229-fceratto.json [production]
16:17 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P90953 and previous config saved to /var/cache/conftool/dbconfig/20260416-161719-fceratto.json [production]
16:15 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2175 (T419961)', diff saved to https://phabricator.wikimedia.org/P90952 and previous config saved to /var/cache/conftool/dbconfig/20260416-161504-fceratto.json [production]
16:14 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
16:14 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T419961)', diff saved to https://phabricator.wikimedia.org/P90951 and previous config saved to /var/cache/conftool/dbconfig/20260416-161432-fceratto.json [production]
16:11 <eevans@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1010.eqiad.wmnet with reason: Bootstrapping — T412830 [production]
16:07 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1248 (T419635)', diff saved to https://phabricator.wikimedia.org/P90950 and previous config saved to /var/cache/conftool/dbconfig/20260416-160710-fceratto.json [production]
16:04 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P90949 and previous config saved to /var/cache/conftool/dbconfig/20260416-160424-fceratto.json [production]
15:54 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P90948 and previous config saved to /var/cache/conftool/dbconfig/20260416-155416-fceratto.json [production]
15:44 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T419961)', diff saved to https://phabricator.wikimedia.org/P90947 and previous config saved to /var/cache/conftool/dbconfig/20260416-154408-fceratto.json [production]
15:35 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2148 (T419961)', diff saved to https://phabricator.wikimedia.org/P90946 and previous config saved to /var/cache/conftool/dbconfig/20260416-153547-fceratto.json [production]
15:35 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
15:35 <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on krb2002.codfw.wmnet with reason: T407726 [production]
15:35 <cgoubert@deploy1003> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
15:35 <cgoubert@deploy1003> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
15:34 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
15:34 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
15:31 <cgoubert@deploy1003> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
15:30 <cgoubert@deploy1003> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
15:29 <cdanis> 💔cdanis@cumin1003.eqiad.wmnet ~ 🕦☕ sudo cumin 'A:swift-fe' 'disable-puppet "cdanis deploy I3aaec0ca T328872"' [production]
15:16 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [production]
15:16 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
15:15 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
15:14 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
15:14 <moritzm> installing sequoia-sqv security updates [production]
15:13 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
15:10 <daniel@deploy1003> Finished scap sync-world: Backport for [[gerrit:1270765|API rate limits: add highlimits-user class (T419796)]] (duration: 10m 47s) [production]
15:04 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [production]
15:03 <daniel@deploy1003> daniel: Continuing with sync [production]
15:01 <daniel@deploy1003> daniel: Backport for [[gerrit:1270765|API rate limits: add highlimits-user class (T419796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
15:00 <root@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mr1-codfw,mr1-codfw IPv6,mr1-codfw.oob with reason: router upgrade [production]
14:59 <daniel@deploy1003> Started scap sync-world: Backport for [[gerrit:1270765|API rate limits: add highlimits-user class (T419796)]] [production]
14:58 <root@cumin2002> DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mr1-codfw IPv6,mr-codfw with reason: router upgrade [production]
14:58 <papaul> ongoing maintenace on mr1-codfw [production]
14:56 <root@cumin2002> DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mr1-codfw IPv6,mr1-codfw.oob,mr-codfw with reason: router upgrade [production]
14:56 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [production]
14:56 <jelto@cumin1003> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host gerrit2002.wikimedia.org [production]
14:56 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
14:54 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
14:54 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
14:52 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
14:45 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org [production]
14:44 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [production]
14:29 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1271895|mc: Use MCROUTER_SERVER values rather than local sidepod for WF cache (T423311)]] (duration: 09m 36s) [production]
14:25 <jforrester@deploy1003> jforrester: Continuing with sync [production]
14:25 <dpogorzelski@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [production]
14:21 <jforrester@deploy1003> jforrester: Backport for [[gerrit:1271895|mc: Use MCROUTER_SERVER values rather than local sidepod for WF cache (T423311)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
14:20 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1271895|mc: Use MCROUTER_SERVER values rather than local sidepod for WF cache (T423311)]] [production]