1-50 of 10000 results (88ms)
2023-12-13 ยง
23:48 <brett@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host acmechief1002.eqiad.wmnet [production]
23:48 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host acmechief1002.eqiad.wmnet with OS bookworm [production]
23:42 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host testhost2001.codfw.wmnet with OS bullseye [production]
23:21 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1107.eqiad.wmnet with OS bookworm [production]
23:17 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply new extra plugins - bking@cumin2002 - T353270 [production]
23:05 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1107.eqiad.wmnet with reason: host reimage [production]
23:02 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1107.eqiad.wmnet with reason: host reimage [production]
22:58 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
22:58 <eevans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:57 <jhuneidi@deploy2002> Finished scap: Backport for [[gerrit:982867|Update wgStatsTarget to port 9125 (T240685)]], [[gerrit:982925|[BC] Enable desktop diff and history pages on mobile (T350181 T353388)]] (duration: 09m 42s) [production]
22:57 <eevans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:54 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bullseye [production]
22:54 <eevans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:53 <eevans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:50 <jhuneidi@deploy2002> jhuneidi and jdlrobson and cwhite: Continuing with sync [production]
22:49 <jhuneidi@deploy2002> jhuneidi and jdlrobson and cwhite: Backport for [[gerrit:982867|Update wgStatsTarget to port 9125 (T240685)]], [[gerrit:982925|[BC] Enable desktop diff and history pages on mobile (T350181 T353388)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:48 <jhuneidi@deploy2002> Started scap: Backport for [[gerrit:982867|Update wgStatsTarget to port 9125 (T240685)]], [[gerrit:982925|[BC] Enable desktop diff and history pages on mobile (T350181 T353388)]] [production]
22:47 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1107.eqiad.wmnet with OS bookworm [production]
22:45 <jhuneidi@deploy2002> Finished scap: Backport for [[gerrit:982835|tests: Use MediaWikiIntegrationTestCase::setGroupPermissions (T353210)]], [[gerrit:982834|Temporarily disable isPreview in Parsoid's rendering]] (duration: 10m 08s) [production]
22:45 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic1107.eqiad.wmnet with OS bookworm [production]
22:45 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1107.eqiad.wmnet with OS bookworm [production]
22:42 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
22:40 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1004.eqiad.wmnet with OS bullseye [production]
22:40 <eevans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:39 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage [production]
22:39 <eevans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - eevans@cumin1001" [production]
22:38 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic1107.eqiad.wmnet with OS bookworm [production]
22:38 <jhuneidi@deploy2002> ssastry and jhuneidi: Continuing with sync [production]
22:38 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1107.eqiad.wmnet with OS bookworm [production]
22:37 <jhuneidi@deploy2002> ssastry and jhuneidi: Backport for [[gerrit:982835|tests: Use MediaWikiIntegrationTestCase::setGroupPermissions (T353210)]], [[gerrit:982834|Temporarily disable isPreview in Parsoid's rendering]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:37 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
22:36 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage [production]
22:35 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host acmechief1002.eqiad.wmnet with OS bookworm [production]
22:35 <jhuneidi@deploy2002> Started scap: Backport for [[gerrit:982835|tests: Use MediaWikiIntegrationTestCase::setGroupPermissions (T353210)]], [[gerrit:982834|Temporarily disable isPreview in Parsoid's rendering]] [production]
22:26 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1004.eqiad.wmnet with reason: host reimage [production]
22:24 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
22:24 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye [production]
22:22 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1004.eqiad.wmnet with reason: host reimage [production]
22:18 <jhuneidi@deploy2002> Finished scap: Backport for [[gerrit:982857|Partially undeploy Reader Demographics 2 survey (T344393)]], [[gerrit:955015|Enable $wgStatsTarget for requests to mwdebug (T240685)]] (duration: 12m 33s) [production]
22:18 <brett@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM acmechief1002.eqiad.wmnet - brett@cumin2002" [production]
22:17 <brett@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM acmechief1002.eqiad.wmnet - brett@cumin2002" [production]
22:16 <brett@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) acmechief1002.eqiad.wmnet on all recursors [production]
22:16 <brett@cumin2002> START - Cookbook sre.dns.wipe-cache acmechief1002.eqiad.wmnet on all recursors [production]
22:16 <brett@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:16 <brett@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM acmechief1002.eqiad.wmnet - brett@cumin2002" [production]
22:15 <brett@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM acmechief1002.eqiad.wmnet - brett@cumin2002" [production]
22:12 <brett@cumin2002> START - Cookbook sre.dns.netbox [production]
22:11 <brett@cumin2002> START - Cookbook sre.ganeti.makevm for new host acmechief1002.eqiad.wmnet [production]
22:11 <jhuneidi@deploy2002> dani and jhuneidi and cwhite: Continuing with sync [production]
22:09 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1004.eqiad.wmnet with OS bullseye [production]