6151-6200 of 10000 results (92ms)
2023-03-02 §
07:37 <elukey@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:37 <elukey@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:36 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host ml-serve1006.eqiad.wmnet with OS bullseye [production]
07:17 <marostegui> Stop MySQL on db2095 T330975 [production]
01:23 <mutante> doc2001 - stopping apache2 to test alerting - active server is doc1002 but should be switched T327973 T330963 [production]
01:08 <mutante> releases2002 - stopping apache2 to test alerting (active server is 1002 but should be switched) T327975 T330960 [production]
00:28 <mutante> planet1002 - stopping apache2 to test alerting (active host is codfw) [production]
2023-03-01 §
23:23 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns1002.wikimedia.org with OS bullseye [production]
23:01 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns1002.wikimedia.org with reason: host reimage [production]
22:56 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns1002.wikimedia.org with reason: host reimage [production]
22:51 <mutante> apt1001 - systemctl reset-failed T328907 [production]
22:45 <mforns@deploy2002> Finished deploy [airflow-dags/analytics@1fb5c4a]: (no justification provided) (duration: 00m 23s) [production]
22:45 <mforns@deploy2002> Started deploy [airflow-dags/analytics@1fb5c4a]: (no justification provided) [production]
22:42 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host dns1002.wikimedia.org with OS bullseye [production]
22:42 <mforns@deploy2002> Finished deploy [airflow-dags/analytics@51e92b1]: (no justification provided) (duration: 00m 21s) [production]
22:42 <mforns@deploy2002> Started deploy [airflow-dags/analytics@51e92b1]: (no justification provided) [production]
21:41 <mforns@deploy2002> Finished deploy [analytics/refinery@d4d723a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d4d723a] (duration: 01m 22s) [production]
21:39 <mforns@deploy2002> Started deploy [analytics/refinery@d4d723a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@d4d723a] [production]
21:39 <mforns@deploy2002> Finished deploy [analytics/refinery@d4d723a] (thin): Regular analytics weekly train THIN [analytics/refinery@d4d723a] (duration: 00m 07s) [production]
21:39 <mforns@deploy2002> Started deploy [analytics/refinery@d4d723a] (thin): Regular analytics weekly train THIN [analytics/refinery@d4d723a] [production]
21:38 <mforns@deploy2002> Finished deploy [analytics/refinery@d4d723a]: Regular analytics weekly train [analytics/refinery@d4d723a] (duration: 10m 55s) [production]
21:37 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2002.wikimedia.org with OS bullseye [production]
21:27 <mforns@deploy2002> Started deploy [analytics/refinery@d4d723a]: Regular analytics weekly train [analytics/refinery@d4d723a] [production]
21:23 <TheresNoTime> closing UTC late backport window [production]
21:18 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2002.wikimedia.org with reason: host reimage [production]
21:16 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns2002.wikimedia.org with reason: host reimage [production]
21:11 <samtar@deploy2002> Finished scap: Backport for [[gerrit:893089|[trwiki] Reverting logo change for Vector 2022 and Vector legacy (T329047)]] (duration: 09m 30s) [production]
21:04 <samtar@deploy2002> superpes and samtar: Backport for [[gerrit:893089|[trwiki] Reverting logo change for Vector 2022 and Vector legacy (T329047)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:02 <samtar@deploy2002> Started scap: Backport for [[gerrit:893089|[trwiki] Reverting logo change for Vector 2022 and Vector legacy (T329047)]] [production]
21:02 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host dns2002.wikimedia.org with OS bullseye [production]
20:43 <zabe> move rev_comment_id migration screens from mwmaint1002 to mwmaint2002 # T275246 [production]
19:47 <brett> re-adding dns3001 to next-hop routing via juniper - T321309 [production]
19:36 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3001.wikimedia.org with OS bullseye [production]
19:12 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns3001.wikimedia.org with reason: host reimage [production]
19:09 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns3001.wikimedia.org with reason: host reimage [production]
18:48 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host dns3001.wikimedia.org with OS bullseye [production]
18:12 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1005.eqiad.wmnet with OS bullseye [production]
18:12 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
18:01 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
18:01 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1003.eqiad.wmnet with OS buster [production]
17:44 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1005.eqiad.wmnet with reason: host reimage [production]
17:41 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1005.eqiad.wmnet with reason: host reimage [production]
17:36 <aqu@deploy2002> Finished deploy [airflow-dags/analytics_test@9568478]: Deploy Airflow upgrade branch for analytics_test (duration: 00m 05s) [production]
17:36 <aqu@deploy2002> Started deploy [airflow-dags/analytics_test@9568478]: Deploy Airflow upgrade branch for analytics_test [production]
17:26 <root@cumin1001> END (PASS) - Cookbook sre.k8s.upgrade-cluster (exit_code=0) Upgrade K8s version: Upgrade to k8s 1.23 [production]
17:24 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1005.eqiad.wmnet with OS bullseye [production]
17:24 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ml-serve1006.eqiad.wmnet with OS bullseye [production]
17:06 <dcaro@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1005'] [production]
17:05 <dcaro@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1005'] [production]
16:56 <dcaro@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1005.eqiad.wmnet with OS bullseye [production]