551-600 of 10000 results (126ms)
2024-11-05 ยง
13:34 <moritzm> imported jenkins 2.479.1 to thirdparty/ci for bullseye-wikimedia T379059 [production]
13:29 <akosiaris@deploy2002> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
13:16 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled [production]
13:16 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled [production]
13:10 <cmooney@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
13:10 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet [production]
13:10 <cmooney@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
13:09 <cmooney@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
13:09 <cmooney@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
13:08 <moritzm> installing php7.4 security updates on remaining non-wikikube servers T378173 [production]
13:03 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet [production]
12:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet [production]
12:50 <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1087424|Revert^2 "temp accounts: Enable temp account creation on second-round pilots" (T378336)]] (duration: 11m 46s) [production]
12:49 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet [production]
12:46 <kharlan@deploy2002> kharlan: Continuing with sync [production]
12:42 <kharlan@deploy2002> kharlan: Backport for [[gerrit:1087424|Revert^2 "temp accounts: Enable temp account creation on second-round pilots" (T378336)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
12:40 <fnegri@cumin1002> END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) [production]
12:39 <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1087424|Revert^2 "temp accounts: Enable temp account creation on second-round pilots" (T378336)]] [production]
12:35 <fnegri@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
12:35 <fnegri@cumin1002> END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=93) [production]
12:35 <fnegri@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
12:34 <fnegri@cumin1002> END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=93) [production]
12:34 <fnegri@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
12:33 <urbanecm> eswiki,x1: `delete from growthexperiments_link_recommendations where gelr_page=10598298;` (to verify updates are flowing in; T378983) [production]
12:33 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1013.eqiad.wmnet [production]
12:33 <urbanecm> mwmaint2002: kill all instances of refreshLinkRecommendation (T378983) [production]
12:32 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1013.eqiad.wmnet [production]
12:28 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1013.eqiad.wmnet [production]
12:23 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1087407|CirrusSearch: Disable updating weighted tags via EventBus (T378983 T377150)]] (duration: 07m 39s) [production]
12:18 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db1125.eqiad.wmnet with reason: testing [production]
12:18 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 6:00:00 on db1125.eqiad.wmnet with reason: testing [production]
12:18 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db2230.codfw.wmnet with reason: testing [production]
12:17 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 6:00:00 on db2230.codfw.wmnet with reason: testing [production]
12:16 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1087407|CirrusSearch: Disable updating weighted tags via EventBus (T378983 T377150)]] [production]
12:10 <jnuche@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 (duration: 07m 43s) [production]
12:04 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1040.eqiad.wmnet to cluster eqiad and group B [production]
12:02 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1040.eqiad.wmnet to cluster eqiad and group B [production]
12:02 <jnuche@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 [production]
12:01 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet [production]
11:57 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet [production]
11:53 <jmm@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1042 [production]
11:53 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.2 refs T375661 [production]
11:53 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es1029 (T376905)', diff saved to https://phabricator.wikimedia.org/P70922 and previous config saved to /var/cache/conftool/dbconfig/20241105-115301-ladsgroup.json [production]
11:52 <jmm@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1042 [production]
11:49 <jmm@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1041 [production]
11:47 <jmm@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1041 [production]
11:47 <jmm@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1040 [production]
11:46 <jmm@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1040 [production]
11:38 <jnuche@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.2 refs T375661 (duration: 36m 28s) [production]
11:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es1029', diff saved to https://phabricator.wikimedia.org/P70921 and previous config saved to /var/cache/conftool/dbconfig/20241105-113754-ladsgroup.json [production]