1-50 of 10000 results (25ms)
2026-04-15 ยง
14:15 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P90792 and previous config saved to /var/cache/conftool/dbconfig/20260415-141535-fceratto.json [production]
14:06 <jforrester@deploy1003> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
14:06 <jforrester@deploy1003> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
14:06 <jforrester@deploy1003> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
14:05 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P90790 and previous config saved to /var/cache/conftool/dbconfig/20260415-140527-fceratto.json [production]
14:04 <jforrester@deploy1003> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
14:04 <jforrester@deploy1003> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
13:56 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw [production]
13:56 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet [production]
13:56 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet [production]
13:55 <samtar@deploy1003> samtar, codenamenoreste: Continuing with sync [production]
13:55 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2157 (T419961)', diff saved to https://phabricator.wikimedia.org/P90789 and previous config saved to /var/cache/conftool/dbconfig/20260415-135519-fceratto.json [production]
13:53 <samtar@deploy1003> samtar, codenamenoreste: Backport for [[gerrit:1271206|lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount (T423102)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:51 <samtar@deploy1003> Started scap sync-world: Backport for [[gerrit:1271206|lbwiki: Set minimum requirement of 10 edits for wgAutoConfirmCount (T423102)]] [production]
13:51 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet [production]
13:50 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet [production]
13:50 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet [production]
13:50 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet [production]
13:47 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2157 (T419961)', diff saved to https://phabricator.wikimedia.org/P90788 and previous config saved to /var/cache/conftool/dbconfig/20260415-134704-fceratto.json [production]
13:46 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
13:45 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet [production]
13:45 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet [production]
13:44 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet [production]
13:44 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet [production]
13:44 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2068.codfw.wmnet with OS bullseye [production]
13:39 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet [production]
13:34 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet [production]
13:34 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet [production]
13:34 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet [production]
13:29 <jmm@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet [production]
13:28 <jmm@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet [production]
13:25 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage [production]
13:21 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1026.eqiad.wmnet [production]
13:21 <eevans@cumin1003> START - Cookbook sre.hosts.remove-downtime for aqs1026.eqiad.wmnet [production]
13:19 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage [production]
13:18 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [production]
13:17 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/wikifeeds: apply [production]
13:17 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance [production]
13:16 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2193 (T419961)', diff saved to https://phabricator.wikimedia.org/P90787 and previous config saved to /var/cache/conftool/dbconfig/20260415-131657-fceratto.json [production]
13:16 <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1270991|Register ArticleGuidance extension and enable in labs (T423295)]] (duration: 12m 02s) [production]
13:12 <kartik@deploy1003> sbisson, kartik: Continuing with sync [production]
13:08 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2221 (T410589)', diff saved to https://phabricator.wikimedia.org/P90786 and previous config saved to /var/cache/conftool/dbconfig/20260415-130849-ladsgroup.json [production]
13:08 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance [production]
13:08 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2218 (T410589)', diff saved to https://phabricator.wikimedia.org/P90785 and previous config saved to /var/cache/conftool/dbconfig/20260415-130836-ladsgroup.json [production]
13:08 <jmm@cumin2002> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw [production]
13:08 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
13:07 <fceratto@cumin1003> DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
13:07 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
13:06 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P90784 and previous config saved to /var/cache/conftool/dbconfig/20260415-130649-fceratto.json [production]
13:06 <kartik@deploy1003> sbisson, kartik: Backport for [[gerrit:1270991|Register ArticleGuidance extension and enable in labs (T423295)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]