1-50 of 10000 results (42ms)
2026-09-23 §
02:08 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:50 <ryankemper@cumin2003> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance [production]
00:46 <ryankemper> [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture [production]
00:45 <ryankemper@cumin2003> START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance [production]
2026-09-22 §
23:23 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [production]
23:23 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet [production]
23:23 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet [production]
23:15 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:45 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:45 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:45 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:36 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:30 <ryankemper> [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts [production]
22:06 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet [production]
22:06 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet [production]
22:06 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:58 <rzl@deploy1003> Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 (duration: 13m 43s) [production]
21:57 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:57 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet [production]
21:57 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:57 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:53 <rzl@deploy1003> rzl: Continuing with deployment [production]
21:50 <rzl@deploy1003> rzl: https://gerrit.wikimedia.org/r/1339694 T437403 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:49 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:47 <rzl@deploy1003> Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 [production]
21:25 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) [production]
21:24 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
21:24 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) [production]
21:19 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet [production]
21:18 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:18 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:10 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet [production]
21:05 <sukhe@cumin1004> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie [production]
20:59 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
20:59 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) [production]
20:59 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [production]
20:55 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) [production]
20:48 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] [production]
20:46 <aqu@deploy1003> Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) [production]
20:45 <bking@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
20:45 <sbisson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1342285|Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) [production]
20:45 <aqu@deploy1003> Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] [production]
20:44 <bking@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
20:44 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
20:43 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
20:40 <sbisson@deploy1003> sbisson: Continuing with deployment [production]
20:40 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet [production]
20:40 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet [production]
20:40 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet [production]