|
2026-03-13
ยง
|
| 10:15 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw |
[production] |
| 10:13 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1008.eqiad.wmnet |
[production] |
| 10:13 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1007.eqiad.wmnet |
[production] |
| 10:12 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tcp-proxy7002.magru.wmnet |
[production] |
| 10:09 |
<jelto@cumin1003> |
conftool action : set/pooled=yes; selector: name=tcp-proxy7001.magru.wmnet |
[production] |
| 10:08 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1007.eqiad.wmnet |
[production] |
| 10:08 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1006.eqiad.wmnet |
[production] |
| 10:07 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7001.magru.wmnet |
[production] |
| 10:03 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tcp-proxy7001.magru.wmnet |
[production] |
| 10:02 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1006.eqiad.wmnet |
[production] |
| 10:02 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1005.eqiad.wmnet |
[production] |
| 10:01 |
<jelto@cumin1003> |
conftool action : set/pooled=no; selector: name=tcp-proxy7001.magru.wmnet |
[production] |
| 09:58 |
<btullis@cumin1003> |
START - Cookbook sre.hosts.reimage for host dse-k8s-worker1019.eqiad.wmnet with OS bookworm |
[production] |
| 09:57 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1005.eqiad.wmnet |
[production] |
| 09:57 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1004.eqiad.wmnet |
[production] |
| 09:51 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1004.eqiad.wmnet |
[production] |
| 09:51 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1003.eqiad.wmnet |
[production] |
| 09:50 |
<javiermonton@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply |
[production] |
| 09:50 |
<javiermonton@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply |
[production] |
| 09:46 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1003.eqiad.wmnet |
[production] |
| 09:46 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1002.eqiad.wmnet |
[production] |
| 09:41 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1002.eqiad.wmnet |
[production] |
| 09:40 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-worker1001.eqiad.wmnet |
[production] |
| 09:39 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:39 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 09:35 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-worker1001.eqiad.wmnet |
[production] |
| 09:35 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-ctrl1002.eqiad.wmnet |
[production] |
| 09:34 |
<filippo@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tools-k8s-ctrl1001.eqiad.wmnet |
[production] |
| 09:34 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:33 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 09:32 |
<moritzm> |
installing Linux 6.1.164 on Bookworm hosts |
[production] |
| 09:30 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-ctrl1002.eqiad.wmnet |
[production] |
| 09:28 |
<filippo@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host tools-k8s-ctrl1001.eqiad.wmnet |
[production] |
| 09:13 |
<elukey> |
upgrade kafka-jumbo and kafka-main to Confluent 7.7 (pre-requisite before being able to upgrade to Trixie) |
[releng] |
| 09:01 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.kafka.change-confluent-distro-version (exit_code=99) Change Confluent distribution for Kafka A:kafka-test-eqiad cluster: Change Confluent distribution. |
[production] |
| 08:37 |
<elukey@cumin1003> |
START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-test-eqiad cluster: Change Confluent distribution. |
[production] |
| 07:56 |
<moritzm> |
installing Linux 6.12.74 on Trixie hosts |
[production] |
| 07:55 |
<moritzm> |
installing 6.12.74 on Trixie hosts |
[production] |
| 02:57 |
<cdobbins@cumin2002> |
conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet [reason: trixie reimaging] |
[production] |
| 02:09 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 08m 18s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 01:41 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS trixie |
[production] |
| 01:37 |
<mutante> |
contint1003/contint2003 - every time(?) we setup machines with puppet using our httpd module and PHP - and puppet runs for the first time we run into the same old issue with "Exec[ensure_present_mod_php" failing and "Considering conflict mpm_worker for mpm_prefork"sudo a2dismod mpm_event". The fix is: 'sudo a2dismod mpm_event' and run puppet again. T418521 |
[production] |
| 01:26 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on contint1003.wikimedia.org with reason: T418521 |
[production] |
| 01:26 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on contint2003.wikimedia.org with reason: T418521 |
[production] |
| 01:23 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint2003.wikimedia.org with reason: setup |
[production] |
| 01:22 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint1003.wikimedia.org with reason: setup |
[production] |
| 01:22 |
<brett@puppetserver1001> |
conftool action : set/pooled=yes; selector: name=cp4047.* |
[production] |
| 01:09 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage |
[production] |
| 01:08 |
<cdobbins@cumin2002> |
conftool action : set/pooled=yes; selector: name=cp4043.ulsfo.wmnet [reason: trixie reimaging] |
[production] |