6851-6900 of 10000 results (94ms)
2024-03-22 ยง
11:55 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
11:55 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
11:52 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
11:52 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
11:39 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
11:39 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:59 <btullis@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:59 <btullis@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:56 <klausman@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
10:55 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
10:54 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:47 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
10:47 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:39 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-worker1168.eqiad.wmnet with reason: Investigating disk errors [production]
10:38 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-worker1168.eqiad.wmnet with reason: Investigating disk errors [production]
10:36 <btullis@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:36 <btullis@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:34 <btullis@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:34 <btullis@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1168.eqiad.wmnet [production]
10:29 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
10:29 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:23 <klausman@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
10:17 <moritzm> uploaded jenkins 2.440.2 to apt.wikimedia.org T360759 [production]
10:16 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:16 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
10:09 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
10:09 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:06 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts apt1001.wikimedia.org [production]
10:06 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:06 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: apt1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
10:04 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
10:04 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:02 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: apt1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:56 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
09:56 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
09:53 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
09:52 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
09:52 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
09:49 <brouberol@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:49 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts apt1001.wikimedia.org [production]
09:47 <brouberol@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:45 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
09:45 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
09:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts apt2001.wikimedia.org [production]
09:39 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:39 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: apt2001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:38 <jnuche@deploy1002> Installation of scap version "4.73.1" completed for 371 hosts [production]
09:37 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: apt2001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:37 <jnuche@deploy1002> Installing scap version "4.73.1" for 371 hosts [production]
09:36 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]