301-350 of 10000 results (48ms)
2023-03-21 ยง
15:52 <jhathaway@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dborch1002.wikimedia.org on all recursors [production]
15:52 <jhathaway@cumin1001> START - Cookbook sre.dns.wipe-cache dborch1002.wikimedia.org on all recursors [production]
15:52 <jhathaway@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
15:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main1005.eqiad.wmnet with OS bullseye [production]
15:51 <jhathaway@cumin1001> START - Cookbook sre.dns.netbox [production]
15:51 <jhathaway@cumin1001> START - Cookbook sre.ganeti.makevm for new host dborch1002.wikimedia.org [production]
15:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:42 <jbond> stop puppet from deploying this further [production]
15:34 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:34 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:34 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:32 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main1005.eqiad.wmnet with reason: host reimage [production]
15:31 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host thanos-fe1004.eqiad.wmnet with OS bullseye [production]
15:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-main1005.eqiad.wmnet with reason: host reimage [production]
15:26 <samtar@deploy2002> Finished scap: Backport for [[gerrit:900828|InitialiseSettings: Set wgAbuseFilterLocallyDisabledGlobalActions (T332521)]] (duration: 09m 11s) [production]
15:22 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:19 <samtar@deploy2002> samtar: Backport for [[gerrit:900828|InitialiseSettings: Set wgAbuseFilterLocallyDisabledGlobalActions (T332521)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
15:17 <samtar@deploy2002> Started scap: Backport for [[gerrit:900828|InitialiseSettings: Set wgAbuseFilterLocallyDisabledGlobalActions (T332521)]] [production]
15:17 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:16 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
15:10 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main1005.eqiad.wmnet with OS bullseye [production]
15:10 <samtar@deploy2002> Finished scap: Backport for [[gerrit:901289|wgAbuseFilterConditionLimit: Set default condition limit to 2000 (T309609)]] (duration: 09m 32s) [production]
15:09 <hnowlan@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [production]
15:02 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1005.eqiad.wmnet with OS bullseye [production]
15:02 <samtar@deploy2002> samtar: Backport for [[gerrit:901289|wgAbuseFilterConditionLimit: Set default condition limit to 2000 (T309609)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
15:02 <hnowlan@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [production]
15:00 <samtar@deploy2002> Started scap: Backport for [[gerrit:901289|wgAbuseFilterConditionLimit: Set default condition limit to 2000 (T309609)]] [production]
14:59 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps1005.eqiad.wmnet [production]
14:51 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps1005.eqiad.wmnet [production]
14:49 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: service=kartotherian,name=maps1005.eqiad.wmnet [production]
14:47 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: service=thumbor,name=maps1005.eqiad.wmnet [production]
14:38 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main1005.eqiad.wmnet with OS bullseye [production]
14:38 <hnowlan> disabling puppet on maps* before merging 760619 [production]
14:37 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1005.eqiad.wmnet with OS bullseye [production]
14:29 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:29 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:27 <elukey@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1005.eqiad.wmnet [production]
14:17 <jnuche@deploy2002> Installing scap version "latest" for 587 hosts [production]
14:15 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:15 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:14 <jnuche@deploy2002> Installing scap version "latest" for 587 hosts [production]
14:11 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
14:11 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
14:10 <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:901588|Growth: Disable GEPersonalizedPraiseEnabled everywhere (T322443)]] (duration: 07m 53s) [production]
14:10 <elukey@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1005.eqiad.wmnet [production]
14:08 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:08 <nfraison@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:05 <elukey@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts kafka-main1005.eqiad.wmnet [production]
14:02 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:901588|Growth: Disable GEPersonalizedPraiseEnabled everywhere (T322443)]] [production]