101-150 of 10000 results (91ms)
2024-08-16 ยง
13:43 <fnegri@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1017.eqiad.wmnet with reason: Reimaging clouddb1017 T365424 [production]
13:41 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s3 [production]
13:41 <fnegri@cumin1002> conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1 [production]
13:26 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd1035.eqiad.wmnet with OS bullseye [production]
12:49 <isaranto@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
11:32 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
11:21 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
10:21 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1007.eqiad.wmnet with OS bullseye [production]
10:19 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
10:16 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
10:14 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1008.eqiad.wmnet with OS bullseye [production]
10:10 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1006.eqiad.wmnet with OS bullseye [production]
10:05 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage [production]
10:02 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage [production]
09:58 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage [production]
09:58 <klausman@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:57 <klausman@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
09:56 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage [production]
09:53 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1010.eqiad.wmnet with reason: host reimage [production]
09:53 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage [production]
09:51 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1009.eqiad.wmnet with reason: host reimage [production]
09:51 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1008.eqiad.wmnet with reason: host reimage [production]
09:51 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1007.eqiad.wmnet with reason: host reimage [production]
09:50 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1006.eqiad.wmnet with reason: host reimage [production]
09:50 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: sync [production]
09:46 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: sync [production]
09:44 <klausman@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
09:43 <klausman@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
09:35 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
09:35 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
09:34 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS bullseye [production]
09:34 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS bullseye [production]
09:33 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS bullseye [production]
09:30 <klausman@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:29 <klausman@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
09:23 <pfischer@deploy1003> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
09:23 <pfischer@deploy1003> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
08:52 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
08:50 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
08:49 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1008.eqiad.wmnet with OS bullseye [production]
08:48 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1007.eqiad.wmnet with OS bullseye [production]
08:47 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1006.eqiad.wmnet with OS bullseye [production]
08:20 <pfischer@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
08:20 <pfischer@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
08:05 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
08:03 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
08:02 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1008.eqiad.wmnet with OS bullseye [production]
08:01 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1007.eqiad.wmnet with OS bullseye [production]
08:00 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1006.eqiad.wmnet with OS bullseye [production]
07:43 <XioNoX> deploy pfw policy update 1723675086 - T372520 [production]