1-50 of 10000 results (78ms)
2023-03-12 §
10:47 <elukey> reset offsets on kafka jumbo for benthos webrequest live (as indicated in https://phabricator.wikimedia.org/T331801#8685569) [production]
07:50 <elukey> restart benthos-webrequest-live on centrallog1002 - T331801 [production]
07:49 <elukey> restart benthos-webrequest-live on centrallog2002 - T331801 [production]
07:49 <elukey> stop and mask benthos-webrequest-live on centrallog1001 - T331801 [production]
2023-03-10 §
22:43 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
22:32 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
22:26 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
22:16 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:24 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:14 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:13 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:03 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
20:43 <milimetric@deploy2002> Finished deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@dd7fc78] (duration: 00m 10s) [production]
20:43 <milimetric@deploy2002> Started deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@dd7fc78] [production]
20:20 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
20:20 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
19:39 <milimetric@deploy2002> Finished deploy [analytics/refinery@898a942] (thin): Special deploy for pageview job migration [analytics/refinery@898a942] (duration: 00m 09s) [production]
19:38 <milimetric@deploy2002> Started deploy [analytics/refinery@898a942] (thin): Special deploy for pageview job migration [analytics/refinery@898a942] [production]
19:38 <milimetric@deploy2002> Finished deploy [analytics/refinery@898a942]: Special deploy for pageview job migration [analytics/refinery@898a942] (duration: 08m 08s) [production]
19:33 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host ms-fe1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:30 <milimetric@deploy2002> Started deploy [analytics/refinery@898a942]: Special deploy for pageview job migration [analytics/refinery@898a942] [production]
19:27 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-fe1013.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:24 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host ms-fe1013.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:23 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:23 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new ms-fe servers - cmjohnson@cumin1001" [production]
19:17 <cmjohnson@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new ms-fe servers - cmjohnson@cumin1001" [production]
19:13 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:11 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2003-dev.codfw.wmnet with OS bullseye [production]
19:11 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:07 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1002.eqiad.wmnet [production]
19:02 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:01 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
19:00 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
19:00 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1002.eqiad.wmnet [production]
18:55 <milimetric@deploy2002> Finished deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@bb9a944] (duration: 00m 12s) [production]
18:55 <milimetric@deploy2002> Started deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@bb9a944] [production]
18:51 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
18:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb2003-dev.codfw.wmnet with reason: host reimage [production]
18:42 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb2003-dev.codfw.wmnet with reason: host reimage [production]
18:35 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb2002-dev.codfw.wmnet with reason: host reimage [production]
18:31 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb2002-dev.codfw.wmnet with reason: host reimage [production]
18:22 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudlb2003-dev.codfw.wmnet with OS bullseye [production]
18:13 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
18:12 <cmooney@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
18:04 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
17:59 <cmooney@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
17:53 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
17:52 <cmooney@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
17:51 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:50 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]