1-50 of 10000 results (1ms)
2026-09-23 §
02:08 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:50 <ryankemper@cumin2003> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance [production]
00:46 <ryankemper> [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture [production]
00:45 <ryankemper@cumin2003> START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance [production]
2026-09-22 §
23:23 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [production]
23:23 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet [production]
23:23 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet [production]
23:15 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:45 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:45 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:45 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:43 <Southparkfan> add wmgRedisLockPassword to PrivateSettings.php - T436480 [deployment-prep]
22:43 <Southparkfan> add wmgRedisLockPassword to PrivateSettings.php - T436480 [releng]
22:40 <Southparkfan> create empty /etc/helmfile-defaults/mediawiki/release on -deploy04 and -deploy06 to unbreak Scap sync-masters - T435393 [deployment-prep]
22:40 <Southparkfan> create empty /etc/helmfile-defaults/mediawiki/release on -deploy04 and -deploy06 to unbreak Scap sync-masters - T435393 [releng]
22:36 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:30 <ryankemper> [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts [production]
22:06 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:06 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet [production]
22:06 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:58 <rzl@deploy1003> Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 (duration: 13m 43s) [production]
21:57 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:57 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:57 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:57 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:53 <rzl@deploy1003> rzl: Continuing with deployment [production]
21:50 <rzl@deploy1003> rzl: https://gerrit.wikimedia.org/r/1339694 T437403 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:49 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:47 <rzl@deploy1003> Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 [production]
21:25 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) [production]
21:24 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
21:24 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) [production]
21:22 <Southparkfan> decommission deployment-cumin-3 - T436470 [deployment-prep]
21:22 <Southparkfan> decommission deployment-cumin-3 - T436470 [releng]
21:20 <Southparkfan> add deployment-deploy06 to scap::dsh::scap_masters and deployment_hosts - T435393 [releng]
21:20 <Southparkfan> add deployment-deploy06 to scap::dsh::scap_masters and deployment_hosts - T435393 [deployment-prep]
21:19 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:18 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:18 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:10 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:05 <sukhe@cumin1004> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie [production]
20:59 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
20:59 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) [production]
20:59 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
20:55 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) [production]
20:53 <Southparkfan> remove puppetserver cherry-pick for T428052, running newer version of HAProxy nowadays [deployment-prep]
20:53 <Southparkfan> remove puppetserver cherry-pick for T428052, running newer version of HAProxy nowadays [releng]
20:49 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [admin]
20:48 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] [production]