251-300 of 10000 results (54ms)
2023-03-21 ยง
20:09 <stevemunene@cumin1001> START - Cookbook sre.ganeti.reimage for host an-test-client1002.eqiad.wmnet with OS bullseye [production]
20:04 <taavi@deploy2002> esanders and taavi and matmarex: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
20:02 <taavi@deploy2002> Started scap: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] [production]
19:52 <stevemunene@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host an-test-client1002.eqiad.wmnet with OS bullseye [production]
19:44 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS bullseye [production]
19:43 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host thanos-fe1004.eqiad.wmnet with OS bullseye [production]
19:41 <jhathaway@cumin1001> END (ERROR) - Cookbook sre.ganeti.reimage (exit_code=97) for host dborch1002.wikimedia.org with OS bullseye [production]
19:17 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS bullseye [production]
19:09 <dancy@deploy2002> Installation of scap version "4.47.1" completed for 587 hosts [production]
19:07 <dancy@deploy2002> Installing scap version "4.47.1" for 587 hosts [production]
19:04 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dborch1002.wikimedia.org with reason: host reimage [production]
19:03 <ebernhardson@deploy2002> Finished deploy [airflow-dags/search@e7b1d0b]: initial deployment of glent dag (duration: 00m 14s) [production]
19:03 <ebernhardson@deploy2002> Started deploy [airflow-dags/search@e7b1d0b]: initial deployment of glent dag [production]
19:01 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dborch1002.wikimedia.org with reason: host reimage [production]
18:52 <jhathaway@cumin1001> START - Cookbook sre.ganeti.reimage for host dborch1002.wikimedia.org with OS bullseye [production]
18:38 <stevemunene@cumin1001> START - Cookbook sre.ganeti.reimage for host an-test-client1002.eqiad.wmnet with OS bullseye [production]
18:36 <dancy@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.1 refs T330207 [production]
18:00 <AndyRussG> update SmashPig config 59a8b2d2 -> 6e651fd [production]
17:48 <jhathaway@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host dborch1002.wikimedia.org [production]
17:40 <joal@deploy2002> Finished deploy [airflow-dags/analytics@e7b1d0b]: Fix analytics HDFSArchiver tasks [airflow-dags/analytics@e7b1d0b] (duration: 00m 11s) [production]
17:39 <joal@deploy2002> Started deploy [airflow-dags/analytics@e7b1d0b]: Fix analytics HDFSArchiver tasks [airflow-dags/analytics@e7b1d0b] [production]
17:25 <stevemunene@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host an-test-client1002.eqiad.wmnet [production]
17:07 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
17:07 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:53 <mutante> sudo cumin -b 4 -s 40 'C:role::cache::text' 'run-puppet-agent' [production]
16:50 <jbond> copy /usr/bin/prometheus-ipmi-exporter from bullseye to buster [production]
16:46 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dborch1002.wikimedia.org on all recursors [production]
16:46 <jhathaway@cumin1001> START - Cookbook sre.dns.wipe-cache dborch1002.wikimedia.org on all recursors [production]
16:46 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:46 <jhathaway@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dborch1002.wikimedia.org - jhathaway@cumin1001" [production]
16:45 <jhathaway@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dborch1002.wikimedia.org - jhathaway@cumin1001" [production]
16:43 <jhathaway@cumin1001> START - Cookbook sre.dns.netbox [production]
16:43 <jhathaway@cumin1001> START - Cookbook sre.ganeti.makevm for new host dborch1002.wikimedia.org [production]
16:33 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host thanos-fe1004.eqiad.wmnet with OS bullseye [production]
16:30 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:30 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:28 <jbond> upload prometheus-ipmi-exporter_1.6.1 to bullseye [production]
16:15 <stevemunene@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-test-client1002.eqiad.wmnet on all recursors [production]
16:15 <stevemunene@cumin1001> START - Cookbook sre.dns.wipe-cache an-test-client1002.eqiad.wmnet on all recursors [production]
16:14 <stevemunene@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:14 <stevemunene@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-test-client1002.eqiad.wmnet - stevemunene@cumin1001" [production]
16:13 <stevemunene@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-test-client1002.eqiad.wmnet - stevemunene@cumin1001" [production]
16:10 <stevemunene@cumin1001> START - Cookbook sre.dns.netbox [production]
16:10 <stevemunene@cumin1001> START - Cookbook sre.ganeti.makevm for new host an-test-client1002.eqiad.wmnet [production]
15:57 <jynus> running from cumin1001: transfer.py --type=decompress dbprov1003.eqiad.wmnet:/srv/backups/snapshots/latest/snapshot.s5.2023-03-20--04-00-30.tar.gz db1145.eqiad.wmnet:/srv/sqldata.s5 [production]
15:53 <jhathaway@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dborch1002.wikimedia.org [production]
15:53 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dborch1002.wikimedia.org on all recursors [production]
15:53 <jhathaway@cumin1001> START - Cookbook sre.dns.wipe-cache dborch1002.wikimedia.org on all recursors [production]
15:53 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:52 <jhathaway@cumin1001> START - Cookbook sre.dns.netbox [production]