101-150 of 10000 results (72ms)
2023-11-22 ยง
20:51 <dzahn@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM planet1003.eqiad.wmnet - dzahn@cumin1001" [production]
20:51 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) planet1003.eqiad.wmnet on all recursors [production]
20:50 <dzahn@cumin1001> START - Cookbook sre.dns.wipe-cache planet1003.eqiad.wmnet on all recursors [production]
20:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM planet1003.eqiad.wmnet - dzahn@cumin1001" [production]
20:50 <dzahn@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM planet1003.eqiad.wmnet - dzahn@cumin1001" [production]
20:47 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
20:47 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host planet1003.eqiad.wmnet [production]
20:45 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1175.eqiad.wmnet with OS bullseye [production]
20:35 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs2008.codfw.wmnet with OS bullseye [production]
20:35 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs2007.codfw.wmnet [production]
20:35 <eevans@cumin1001> START - Cookbook sre.hosts.remove-downtime for aqs2007.codfw.wmnet [production]
20:33 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2007.codfw.wmnet with OS bullseye [production]
20:29 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T348183)', diff saved to https://phabricator.wikimedia.org/P53735 and previous config saved to /var/cache/conftool/dbconfig/20231122-202947-arnaudb.json [production]
20:14 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P53734 and previous config saved to /var/cache/conftool/dbconfig/20231122-201441-arnaudb.json [production]
20:14 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2007.codfw.wmnet with reason: host reimage [production]
20:11 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2007.codfw.wmnet with reason: host reimage [production]
19:59 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P53733 and previous config saved to /var/cache/conftool/dbconfig/20231122-195934-arnaudb.json [production]
19:58 <ejegg> standalone (payments listener) SmashPig upgraded from b867e553 to f24afba3 [production]
19:56 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs2007.codfw.wmnet with OS bullseye [production]
19:55 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs2006.codfw.wmnet [production]
19:55 <eevans@cumin1001> START - Cookbook sre.hosts.remove-downtime for aqs2006.codfw.wmnet [production]
19:48 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2006.codfw.wmnet with OS bullseye [production]
19:44 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T348183)', diff saved to https://phabricator.wikimedia.org/P53732 and previous config saved to /var/cache/conftool/dbconfig/20231122-194428-arnaudb.json [production]
19:37 <ejegg> fundraising civicrm upgraded from 3c5db93b to f3de1778 [production]
19:36 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1160.eqiad.wmnet with OS bullseye [production]
19:25 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2006.codfw.wmnet with reason: host reimage [production]
19:22 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2006.codfw.wmnet with reason: host reimage [production]
19:06 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs2006.codfw.wmnet with OS bullseye [production]
19:05 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs2005.codfw.wmnet [production]
19:05 <eevans@cumin1001> START - Cookbook sre.hosts.remove-downtime for aqs2005.codfw.wmnet [production]
18:58 <ejegg> standalone SmashPig upgraded from c5b12dc3 to b867e553 [production]
18:57 <cstone> payments-wiki upgraded from 714552c5 to f02f8653 [production]
18:50 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
18:16 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1160.eqiad.wmnet with OS bullseye [production]
18:03 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2005.codfw.wmnet with OS bullseye [production]
17:45 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [production]
17:42 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [production]
17:36 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1175'] [production]
17:29 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1175'] [production]
17:28 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
17:28 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
17:27 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-worker1175.mgmt.eqiad.wmnet with reboot policy FORCED [production]
17:27 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs2005.codfw.wmnet with OS bullseye [production]
17:27 <hnowlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
17:27 <hnowlan@deploy2002> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
17:26 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs2004.codfw.wmnet [production]
17:26 <eevans@cumin1001> START - Cookbook sre.hosts.remove-downtime for aqs2004.codfw.wmnet [production]
17:25 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
17:25 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]