2151-2200 of 10000 results (79ms)
2023-08-16 ยง
14:12 <urbanecm@deploy1002> urbanecm: Backport for [[gerrit:949184|Revert "Growth: Temporarily disable link-recommendation frontend" (T344034)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
14:12 <jmm@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
14:10 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949184|Revert "Growth: Temporarily disable link-recommendation frontend" (T344034)]] [production]
14:10 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
14:10 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netflow3003.esams.wmnet on all recursors [production]
14:10 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache netflow3003.esams.wmnet on all recursors [production]
14:10 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:10 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow3003.esams.wmnet - jmm@cumin2002" [production]
14:09 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow3003.esams.wmnet - jmm@cumin2002" [production]
14:06 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s) [production]
14:05 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
14:05 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host netflow3003.esams.wmnet [production]
14:03 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1101.eqiad.wmnet with reason: host reimage [production]
14:00 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1101.eqiad.wmnet with reason: host reimage [production]
14:00 <urbanecm@deploy1002> urbanecm and d3r1ck01: Continuing with sync [production]
13:53 <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl [production]
13:52 <sukhe> restart pybal on lvs3008 [production]
13:52 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] [production]
13:51 <sukhe> running authdns-update for CR 949113 [production]
13:51 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949512|Add blkwiktionary logo (T344310)]], [[gerrit:949513|add suwikisource logo (T344314)]] (duration: 10m 37s) [production]
13:46 <sukhe> running homer on asw1-b*27-esams* for CR 949100: T329219 [production]
13:44 <urbanecm@deploy1002> urbanecm and anzx: Continuing with sync [production]
13:43 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1101.eqiad.wmnet with OS bullseye [production]
13:42 <urbanecm@deploy1002> urbanecm and anzx: Backport for [[gerrit:949512|Add blkwiktionary logo (T344310)]], [[gerrit:949513|add suwikisource logo (T344314)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
13:41 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949512|Add blkwiktionary logo (T344310)]], [[gerrit:949513|add suwikisource logo (T344314)]] [production]
13:40 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949030|testwikidatawiki: always show MUL in Termbox (T343409)]] (duration: 09m 43s) [production]
13:33 <urbanecm@deploy1002> migr and urbanecm: Continuing with sync [production]
13:32 <urbanecm@deploy1002> migr and urbanecm: Backport for [[gerrit:949030|testwikidatawiki: always show MUL in Termbox (T343409)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
13:30 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949030|testwikidatawiki: always show MUL in Termbox (T343409)]] [production]
13:29 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:948094|GrowthExperiments: enable add a link in 11th round of wikis (T308136)]] (duration: 11m 32s) [production]
13:29 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
13:28 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
13:23 <urbanecm@deploy1002> sgimeno and urbanecm: Continuing with sync [production]
13:19 <urbanecm@deploy1002> sgimeno and urbanecm: Backport for [[gerrit:948094|GrowthExperiments: enable add a link in 11th round of wikis (T308136)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
13:18 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:948094|GrowthExperiments: enable add a link in 11th round of wikis (T308136)]] [production]
12:59 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:59 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:58 <urbanecm> mwscript extensions/GrowthExperiments/maintenance/revalidateLinkRecommendations.php --wiki=ruwiki --olderThan=1651960800 --verbose # T344034 [production]
12:54 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949510|Growth: Temporarily disable link-recommendation frontend (T344034)]] (duration: 08m 04s) [production]
12:53 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:53 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:52 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:51 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:50 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet [production]
12:46 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949510|Growth: Temporarily disable link-recommendation frontend (T344034)]] [production]
12:42 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet [production]
12:39 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:39 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti3008.esams.wmnet to cluster esams02 and group BW27 [production]
12:37 <jiji@deploy1002> helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply [production]
12:37 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti3008.mgmt.esams.wmnet with reboot policy GRACEFUL [production]