3951-4000 of 10000 results (47ms)
2021-02-12 ยง
19:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw1358.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw1358.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw1353.eqiad.wmnet with reason: OS upgrade [production]
19:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw1353.eqiad.wmnet with reason: OS upgrade [production]
19:46 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mwdebug1003.eqiad.wmnet with reason: OS upgrade [production]
19:46 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mwdebug1003.eqiad.wmnet with reason: OS upgrade [production]
19:43 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: roll back commonswiki to 1.36.0-wmf.27 due to T274589 [production]
19:42 <mutante> mwdebug2001 now on buster - mwdebug1003 rebooting and reimaging to stretch [production]
19:38 <milimetric@deploy1001> Finished deploy [analytics/refinery@e0c09a2] (thin): Fix for mediarequest per file cassandra job - 2 (duration: 00m 06s) [production]
19:38 <milimetric@deploy1001> Started deploy [analytics/refinery@e0c09a2] (thin): Fix for mediarequest per file cassandra job - 2 [production]
19:38 <milimetric@deploy1001> Finished deploy [analytics/refinery@e0c09a2]: Fix for mediarequest per file cassandra job - 2 (duration: 11m 01s) [production]
19:34 <twentyafterfour> Train status: Rolling back commonswiki to wmf.27 due to T274589 (refs T271344) [production]
19:29 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1358.eqiad.wmnet with reason: REIMAGE [production]
19:28 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1353.eqiad.wmnet with reason: REIMAGE [production]
19:28 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on aqs1015.eqiad.wmnet with reason: REIMAGE [production]
19:27 <milimetric@deploy1001> Started deploy [analytics/refinery@e0c09a2]: Fix for mediarequest per file cassandra job - 2 [production]
19:24 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1358.eqiad.wmnet with reason: REIMAGE [production]
19:23 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1353.eqiad.wmnet with reason: REIMAGE [production]
19:23 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1015.eqiad.wmnet with reason: REIMAGE [production]
19:20 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1013.eqiad.wmnet with reason: REIMAGE [production]
19:18 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1013.eqiad.wmnet with reason: REIMAGE [production]
19:18 <milimetric@deploy1001> Finished deploy [analytics/refinery@366962f]: Fix for mediarequest per file cassandra job (duration: 11m 58s) [production]
19:17 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1012.eqiad.wmnet with reason: REIMAGE [production]
19:15 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1011.eqiad.wmnet with reason: REIMAGE [production]
19:15 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1012.eqiad.wmnet with reason: REIMAGE [production]
19:13 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1010.eqiad.wmnet with reason: REIMAGE [production]
19:13 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1011.eqiad.wmnet with reason: REIMAGE [production]
19:11 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1010.eqiad.wmnet with reason: REIMAGE [production]
19:06 <milimetric@deploy1001> Started deploy [analytics/refinery@366962f]: Fix for mediarequest per file cassandra job [production]
19:02 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mwdebug2001.codfw.wmnet with reason: OS upgrade [production]
19:02 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mwdebug2001.codfw.wmnet with reason: OS upgrade [production]
19:02 <mutante> rebooting and reimaging mwdebug2001 to buster T274023 [production]
18:35 <mutante> mwdebug2002 now a buster VM; you can find a .tar.gz in your home dir with the contents of your previous home [production]
18:30 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@98264b8]: airflow: review and correct usage of catchup=False (duration: 03m 10s) [production]
18:27 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@98264b8]: airflow: review and correct usage of catchup=False [production]
17:33 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.reboot-workers (exit_code=0) for Presto analytics cluster: Reboot Presto nodes - elukey@cumin1001 [production]
17:23 <bblack> cp*: re-enabling puppet after successful agent run on one host as a test! [production]
17:13 <bblack> cp*: disable puppet ahead of https://gerrit.wikimedia.org/r/c/operations/puppet/+/663845 [production]
17:08 <elukey@cumin1001> START - Cookbook sre.presto.reboot-workers for Presto analytics cluster: Reboot Presto nodes - elukey@cumin1001 [production]
17:01 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1005.eqiad.wmnet [production]
16:48 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1005.eqiad.wmnet [production]
16:45 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-ui1001.eqiad.wmnet [production]
16:43 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-ui1001.eqiad.wmnet [production]
16:43 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1001.eqiad.wmnet [production]
16:38 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-client1001.eqiad.wmnet [production]
16:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-druid1001.eqiad.wmnet [production]
16:26 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-druid1001.eqiad.wmnet [production]
16:12 <mforns@deploy1001> Finished deploy [analytics/refinery@9cd1297] (hadoop-test): Fix for data quality alarms after BigTop migration TEST [analytics/refinery@9cd129764edbac04b192c922ec0a975bc47455a5] (duration: 04m 05s) [production]
16:11 <hnowlan> joining maps2007 to cassandra cluster [production]
16:08 <mforns@deploy1001> Started deploy [analytics/refinery@9cd1297] (hadoop-test): Fix for data quality alarms after BigTop migration TEST [analytics/refinery@9cd129764edbac04b192c922ec0a975bc47455a5] [production]