1951-2000 of 10000 results (75ms)
2022-12-14 §
07:28 <phedenskog@deploy1002> Started deploy [performance/navtiming@7ba179f]: (no justification provided) [production]
01:22 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2037.codfw.wmnet with OS bullseye [production]
01:18 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2036.codfw.wmnet with OS bullseye [production]
01:06 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2037.codfw.wmnet with reason: host reimage [production]
01:04 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2036.codfw.wmnet with reason: host reimage [production]
01:02 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2037.codfw.wmnet with reason: host reimage [production]
01:01 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2036.codfw.wmnet with reason: host reimage [production]
00:46 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2037.codfw.wmnet with OS bullseye [production]
00:45 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2036.codfw.wmnet with OS bullseye [production]
2022-12-13 §
22:57 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:867618|Account for syntax errors in closest selector (T325113)]] (duration: 08m 29s) [production]
22:50 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:867618|Account for syntax errors in closest selector (T325113)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
22:48 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:867618|Account for syntax errors in closest selector (T325113)]] [production]
22:42 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:867617|Account for syntax errors in closest selector (T325113)]] (duration: 09m 20s) [production]
22:40 <dzahn@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "parse1002: failed -> active - dzahn@cumin2002" [production]
22:40 <mutante> netbox: set parse1002 status: failed -> active in web UI; ran cookbook 'sre.puppet.sync-netbox-hiera' to get data in sync - T324949 [production]
22:38 <dzahn@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "parse1002: failed -> active - dzahn@cumin2002" [production]
22:35 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:34 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:867617|Account for syntax errors in closest selector (T325113)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
22:34 <dzahn@cumin2002> START - Cookbook sre.dns.netbox [production]
22:33 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:867617|Account for syntax errors in closest selector (T325113)]] [production]
22:23 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2035.codfw.wmnet with OS bullseye [production]
22:14 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2033.codfw.wmnet with OS bullseye [production]
22:12 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash2034.codfw.wmnet with OS bullseye [production]
22:10 <aqu@deploy1002> Finished deploy [analytics/refinery@66736e1] (hadoop-test): HDFS FSImage conversion to XML script TEST [analytics/refinery@66736e1] (duration: 01m 11s) [production]
22:09 <aqu@deploy1002> Started deploy [analytics/refinery@66736e1] (hadoop-test): HDFS FSImage conversion to XML script TEST [analytics/refinery@66736e1] [production]
22:08 <aqu@deploy1002> Finished deploy [analytics/refinery@66736e1] (thin): HDFS FSImage conversion to XML script THIN [analytics/refinery@66736e1] (duration: 00m 07s) [production]
22:08 <aqu@deploy1002> Started deploy [analytics/refinery@66736e1] (thin): HDFS FSImage conversion to XML script THIN [analytics/refinery@66736e1] [production]
22:06 <aqu@deploy1002> Finished deploy [analytics/refinery@66736e1]: HDFS FSImage conversion to XML script [analytics/refinery@66736e1] (duration: 26m 32s) [production]
21:53 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2033.codfw.wmnet with reason: host reimage [production]
21:52 <reedy@deploy1002> Synchronized wmf-config/CommonSettings.php: extension distributor updates (duration: 06m 50s) [production]
21:50 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2034.codfw.wmnet with reason: host reimage [production]
21:48 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2033.codfw.wmnet with reason: host reimage [production]
21:47 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2034.codfw.wmnet with reason: host reimage [production]
21:43 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2035.codfw.wmnet with reason: host reimage [production]
21:41 <kindrobot> Finishing UTC late backport window [production]
21:40 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:863228|Start writing to cul_actor everywhere (T233004)]] (duration: 18m 47s) [production]
21:40 <aqu@deploy1002> Started deploy [analytics/refinery@66736e1]: HDFS FSImage conversion to XML script [analytics/refinery@66736e1] [production]
21:39 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2035.codfw.wmnet with reason: host reimage [production]
21:35 <aqu> Deploying analytics/refinery (HDFS FSImage conversion to XML script) [production]
21:32 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2033.codfw.wmnet with OS bullseye [production]
21:30 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2034.codfw.wmnet with OS bullseye [production]
21:23 <kindrobot@deploy1002> kindrobot and zabe: Backport for [[gerrit:863228|Start writing to cul_actor everywhere (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
21:23 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash2035.codfw.wmnet with OS bullseye [production]
21:21 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:863228|Start writing to cul_actor everywhere (T233004)]] [production]
21:17 <samtar@deploy1002> Finished scap: Backport for [[gerrit:867233|Child elements also trigger previews (T325007)]] (duration: 09m 38s) [production]
21:09 <samtar@deploy1002> samtar and jdlrobson: Backport for [[gerrit:867233|Child elements also trigger previews (T325007)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
21:08 <samtar@deploy1002> Started scap: Backport for [[gerrit:867233|Child elements also trigger previews (T325007)]] [production]
20:20 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge elasticsearch and plugin upgrade - ryankemper@cumin1001 - T322776 [production]
20:17 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge elasticsearch and plugin upgrade - ryankemper@cumin1001 - T322776 [production]
20:17 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on relforge[1003-1004].eqiad.wmnet with reason: Rolling restart [production]