751-800 of 10000 results (26ms)
2026-11-16 ยง
18:10 <volans@cumin1004> START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org [production]
18:07 <taavi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org [production]
18:04 <volans@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org [production]
18:01 <taavi@cumin1004> START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org [production]
18:00 <taavi@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org [production]
17:58 <volans@cumin1004> START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org [production]
17:58 <volans@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org [production]
17:54 <taavi@cumin1004> START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org [production]
17:51 <volans@cumin1004> START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org [production]
17:49 <ladsgroup@deploy2003> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
17:49 <ladsgroup@deploy2003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
17:47 <sukhe@cumin1004> END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:eqsin and A:hcaptcha-proxy [production]
17:46 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1005.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [production]
17:46 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet [production]
17:46 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet [production]
17:44 <sukhe@cumin1004> START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and not A:eqsin and A:hcaptcha-proxy [production]
17:40 <btullis@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet [production]
17:36 <sukhe@cumin1004> START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:eqsin and A:hcaptcha-proxy [production]
17:36 <sukhe@cumin1004> END (ERROR) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=97) rolling reboot on A:hcaptcha-proxy or A:eqsin and A:hcaptcha-proxy [production]
17:36 <sukhe@cumin1004> START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy or A:eqsin and A:hcaptcha-proxy [production]
17:35 <btullis@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet [production]
17:35 <btullis@cumin1004> START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1005.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [production]
17:33 <ladsgroup@deploy2003> helmfile [eqiad] DONE helmfile.d/services/thumbor: sync [production]
17:33 <ladsgroup@deploy2003> helmfile [eqiad] START helmfile.d/services/thumbor: sync [production]
17:32 <ladsgroup@deploy2003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
17:31 <ladsgroup@deploy2003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
17:31 <sukhe@cumin1004> START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough [production]
17:30 <sukhe@cumin1004> START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy or A:eqiad or A:ulsfo or A:drmrs or A:magru and A:hcaptcha-proxy [production]
17:25 <ladsgroup@deploy2003> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
17:25 <ladsgroup@deploy2003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
17:23 <sukhe@cumin1004> END (FAIL) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=1) rolling reboot on A:wikidough [production]
17:23 <ladsgroup@deploy2003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
17:23 <jgiannelos@deploy2003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
17:22 <jgiannelos@deploy2003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
17:22 <jgiannelos@deploy2003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
17:22 <jgiannelos@deploy2003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
17:22 <ladsgroup@deploy2003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
17:22 <volans> rebooting bast7002 [production]
17:21 <ladsgroup@deploy2003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
17:19 <sukhe@cumin1004> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders (exit_code=0) rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw) [production]
17:00 <sukhe@cumin1004> START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough [production]
17:00 <sukhe@cumin1004> START - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw) [production]
16:58 <volans> rebooting gitlab2002 from console [production]
16:55 <jhathaway@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: new-kernel [production]
16:55 <jhathaway@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out2001.wikimedia.org with reason: new-kernel [production]
16:53 <jhathaway@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: new-kernel [production]
16:53 <volans> rebooting gitlab1003 from console [production]
16:53 <jhathaway@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel [production]
16:52 <jhathaway@cumin1004> DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel [production]
16:29 <tappof@deploy2003> helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [production]