5751-5800 of 10000 results (70ms)
2022-06-09 ยง
17:48 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
17:46 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [production]
17:44 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
17:44 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
17:40 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti3002.esams.wmnet with OS bullseye [production]
17:39 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
17:34 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1040.eqiad.wmnet [production]
17:32 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1064.eqiad.wmnet with OS bullseye [production]
17:29 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1040.eqiad.wmnet [production]
17:23 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti3002.esams.wmnet with reason: host reimage [production]
17:18 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti3002.esams.wmnet with reason: host reimage [production]
17:16 <mbsantos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
17:15 <mbsantos@deploy1002> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
17:14 <mbsantos@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
17:13 <mbsantos@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
17:12 <mbsantos@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
17:12 <mbsantos@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
17:01 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host ganeti3002.esams.wmnet with OS bullseye [production]
16:52 <dancy@deploy1002> rebuilt and synchronized wikiversions files: testing [production]
16:43 <btullis@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs: Rolling AQS Cassandra cluster to pick up new encryption settings - btullis@cumin1001 [production]
16:17 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (3 nodes at a time) for ElasticSearch cluster search_codfw: bullseye upgrade - bking@cumin1001 - T289135 [production]
16:14 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2054.codfw.wmnet with OS bullseye [production]
16:10 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2054.codfw.wmnet with OS bullseye [production]
16:09 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
16:09 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1039.eqiad.wmnet [production]
16:04 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1039.eqiad.wmnet [production]
16:00 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
16:00 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2036.codfw.wmnet with OS bullseye [production]
15:58 <robh> ganeti3002 rebooting into firmware update then reimage via T308238 [production]
15:57 <btullis@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs: Rolling AQS Cassandra cluster to pick up new encryption settings - btullis@cumin1001 [production]
15:53 <moritzm> installing curl security updates [production]
15:52 <pt1979@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1142.eqiad.wmnet with OS buster [production]
15:46 <XioNoX> set cache "pass" to netbox-exports [production]
15:43 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:38 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2036.codfw.wmnet with reason: host reimage [production]
15:35 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2036.codfw.wmnet with reason: host reimage [production]
15:19 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2036.codfw.wmnet with OS bullseye [production]
15:15 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (3 nodes at a time) for ElasticSearch cluster search_codfw: bullseye upgrade - bking@cumin1001 - T289135 [production]
15:07 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1063.eqiad.wmnet with OS bullseye [production]
15:03 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
15:03 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:59 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:56 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS buster [production]
14:53 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:48 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1018.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:44 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:35 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host aqs1018.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:29 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1063.eqiad.wmnet with reason: host reimage [production]
14:26 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1063.eqiad.wmnet with reason: host reimage [production]
14:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on webperf2002.codfw.wmnet with reason: Migration to new Bullseye nodes [production]