3351-3400 of 10000 results (86ms)
2023-12-11 ยง
14:18 <arnaudb@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:18 <arnaudb@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1138.eqiad.wmnet - arnaudb@cumin1001" [production]
14:17 <samtar@deploy2002> samtar and anzx: Continuing with sync [production]
14:16 <arnaudb@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1138.eqiad.wmnet - arnaudb@cumin1001" [production]
14:15 <samtar@deploy2002> samtar and anzx: Backport for [[gerrit:981726|hewikivoyage: update vector 2022 wordmark and tagline (T351981)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:14 <samtar@deploy2002> Started scap: Backport for [[gerrit:981726|hewikivoyage: update vector 2022 wordmark and tagline (T351981)]] [production]
14:11 <samtar@deploy2002> Finished scap: Backport for [[gerrit:979986|Enable read new on group0 wikis (T341829)]] (duration: 07m 57s) [production]
14:05 <samtar@deploy2002> samtar and dreamyjazz: Continuing with sync [production]
14:05 <samtar@deploy2002> samtar and dreamyjazz: Backport for [[gerrit:979986|Enable read new on group0 wikis (T341829)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:03 <samtar@deploy2002> Started scap: Backport for [[gerrit:979986|Enable read new on group0 wikis (T341829)]] [production]
13:59 <arnaudb@cumin1001> START - Cookbook sre.dns.netbox [production]
13:58 <arnaudb@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:56 <arnaudb@cumin1001> START - Cookbook sre.dns.netbox [production]
13:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
13:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
13:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
13:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
13:27 <arnaudb@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db1138.eqiad.wmnet [production]
13:26 <arnaudb@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:25 <arnaudb@cumin1001> START - Cookbook sre.dns.netbox [production]
13:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'decommission db1138', diff saved to https://phabricator.wikimedia.org/P54328 and previous config saved to /var/cache/conftool/dbconfig/20231211-132250-arnaudb.json [production]
13:20 <arnaudb@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1138.eqiad.wmnet [production]
13:17 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: decomission pre downtime [production]
13:17 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: decomission pre downtime [production]
13:13 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
13:12 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
13:05 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
13:04 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
12:57 <claime> Rebuilding production-images for python3-build-bookworm - T352733 [production]
12:12 <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:981734|Revert "Growth: Enable Welcome survey user research for ar/en/es" (T351266)]] (duration: 08m 20s) [production]
12:11 <brouberol> Adding spark-history(-test).svc.eqiad.wmnet CNAMEs pointing to k8s-ingress-dse.svc.eqiad.wmnet. - T352639 [production]
12:05 <urbanecm@deploy2002> urbanecm: Continuing with sync [production]
12:05 <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:981734|Revert "Growth: Enable Welcome survey user research for ar/en/es" (T351266)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
12:03 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:981734|Revert "Growth: Enable Welcome survey user research for ar/en/es" (T351266)]] [production]
11:20 <vgutierrez> rolling restart of pybal on lvs3010 and lvs3008 effectively enabling IPIP encapsulation on ncredir@esams - T351069 [production]
11:18 <claime> sudo confctl --object-type discovery select 'name=eqiad,dnsdisc=k8s-ingress-dse' set/pooled=true - T352639 [production]
11:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
11:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
11:12 <brouberol> Add discovery records for the k8s-ingress-dse LVS service - T352639 [production]
10:55 <dcausse> (properly) restarting blazegraph on wdqs1006 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
10:54 <cgoubert@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[1019-1020].eqiad.wmnet} and A:lvs (T352639) [production]
10:50 <cgoubert@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[1019-1020].eqiad.wmnet} and A:lvs (T352639) [production]
10:46 <claime> Running puppet on O:lvs::balancer - T352639 [production]
10:45 <claime> Disabling puppet on O:lvs::balancer - T352639 [production]
10:42 <elukey@deploy2002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: sync [production]
10:42 <elukey@deploy2002> helmfile [codfw] START helmfile.d/services/recommendation-api: sync [production]
10:42 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: sync [production]
10:38 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/recommendation-api: sync [production]
10:38 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/recommendation-api: sync [production]
10:38 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/recommendation-api: sync [production]