1-50 of 10000 results (22ms)
2026-03-17 ยง
23:44 <btullis@cumin1003> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]
23:38 <btullis@cumin1003> END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [production]
22:55 <brett@puppetserver1001> conftool action : set/pooled=yes; selector: name=cp3081.* [production]
22:50 <James_F> Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs [releng]
22:20 <cdobbins@cumin2002> conftool action : set/pooled=yes; selector: name=cp3073.esams.wmnet [reason: trixie reimaging] [production]
22:15 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3073.esams.wmnet with OS trixie [production]
22:11 <cdobbins@cumin2002> conftool action : set/pooled=yes; selector: name=cp3072.esams.wmnet [reason: trixie reimaging] [production]
22:10 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3072.esams.wmnet with OS trixie [production]
22:05 <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on releases1003.eqiad.wmnet with reason: T420246 [production]
22:05 <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on releases2003.codfw.wmnet with reason: T420246 [production]
21:48 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3073.esams.wmnet with reason: host reimage [production]
21:44 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage [production]
21:41 <cdobbins@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3073.esams.wmnet with reason: host reimage [production]
21:39 <cdobbins@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage [production]
21:38 <ryankemper> T411568 Failed back HDFS NameNode from an-master1004 to an-master1003; cluster back to original active/standby configuration [production]
21:33 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b) [tools.cluebotng-review]
21:27 <James_F> Zuul: search: Update opensearch plugins for Java 11/17, for T420407 [releng]
21:15 <cdobbins@cumin2002> START - Cookbook sre.hosts.reimage for host cp3073.esams.wmnet with OS trixie [production]
21:14 <cdobbins@cumin2002> conftool action : set/pooled=no; selector: name=cp3073.esams.wmnet [reason: trixie reimaging] [production]
21:14 <cdobbins@cumin2002> START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS trixie [production]
21:14 <cdobbins@cumin2002> conftool action : set/pooled=no; selector: name=cp3072.esams.wmnet [reason: trixie reimaging] [production]
21:13 <cdobbins@cumin2002> conftool action : set/pooled=yes; selector: name=cp3070.esams.wmnet [reason: trixie reimaging] [production]
21:13 <cdobbins@cumin2002> conftool action : set/pooled=yes; selector: name=cp3071.esams.wmnet [reason: trixie reimaging] [production]
21:09 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3070.esams.wmnet with OS trixie [production]
21:05 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3071.esams.wmnet with OS trixie [production]
20:59 <alexsanford@deploy2002> Finished scap sync-world: Backport for [[gerrit:1254280|Remove notice from login form in popup mode (T418534)]] (duration: 07m 32s) [production]
20:56 <alexsanford@deploy2002> alexsanford: Continuing with sync [production]
20:54 <alexsanford@deploy2002> alexsanford: Backport for [[gerrit:1254280|Remove notice from login form in popup mode (T418534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:52 <alexsanford@deploy2002> Started scap sync-world: Backport for [[gerrit:1254280|Remove notice from login form in popup mode (T418534)]] [production]
20:48 <btullis@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dse-k8s-worker1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:43 <btullis@cumin1003> START - Cookbook sre.hosts.provision for host dse-k8s-worker1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:43 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3070.esams.wmnet with reason: host reimage [production]
20:40 <btullis@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm [production]
20:40 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1015.eqiad.wmnet with OS bookworm [production]
20:38 <ryankemper> T411568 failed over HDFS NameNode from an-master1003 to an-master1004, then rebooted `an-master1003` [production]
20:38 <ryankemper> T411568 rebooted `an-coord1003`, `an-coord1004`, `an-tool1007`, `an-tool1008`, `an-tool1011`, `an-web1001` [production]
20:38 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3071.esams.wmnet with reason: host reimage [production]
20:34 <cdobbins@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3070.esams.wmnet with reason: host reimage [production]
20:34 <cdobbins@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3071.esams.wmnet with reason: host reimage [production]
20:31 <catrope@deploy2002> Finished scap sync-world: Backport for [[gerrit:1254301|Passwordless login: Don't display conditional auth errors]], [[gerrit:1254302|Passwordless login: Don't display conditional auth errors]] (duration: 08m 56s) [production]
20:27 <catrope@deploy2002> catrope: Continuing with sync [production]
20:24 <catrope@deploy2002> catrope: Backport for [[gerrit:1254301|Passwordless login: Don't display conditional auth errors]], [[gerrit:1254302|Passwordless login: Don't display conditional auth errors]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:22 <catrope@deploy2002> Started scap sync-world: Backport for [[gerrit:1254301|Passwordless login: Don't display conditional auth errors]], [[gerrit:1254302|Passwordless login: Don't display conditional auth errors]] [production]
20:20 <bd808> Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 (T415021) [releng]
20:16 <ryankemper> T411568 rebooted `an-test-master1002`, `an-test-master1003`, `an-test-master1004`, `archiva1002` [production]
20:12 <aude@deploy2002> Finished scap sync-world: Backport for [[gerrit:1251309|Set wgReadingListsBetaDefaultForNewAccountsAfter for beta cluster (T419163)]] (duration: 08m 53s) [production]
20:09 <cdobbins@cumin2002> START - Cookbook sre.hosts.reimage for host cp3071.esams.wmnet with OS trixie [production]
20:09 <cdobbins@cumin2002> START - Cookbook sre.hosts.reimage for host cp3070.esams.wmnet with OS trixie [production]
20:08 <aude@deploy2002> aude: Continuing with sync [production]
20:08 <cdobbins@cumin2002> conftool action : set/pooled=no; selector: name=cp3070.esams.wmnet [reason: trixie reimaging] [production]