3251-3300 of 10000 results (78ms)
2022-09-02 §
07:17 <dcausse> restarting blazegraph on wdqs1016 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
05:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1119 to clone db1107 T316870', diff saved to https://phabricator.wikimedia.org/P33739 and previous config saved to /var/cache/conftool/dbconfig/20220902-054405-root.json [production]
05:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2149 T316494 ', diff saved to https://phabricator.wikimedia.org/P33738 and previous config saved to /var/cache/conftool/dbconfig/20220902-052841-marostegui.json [production]
2022-09-01 §
20:51 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:50 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:50 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:49 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:828616|Remove Vector grid config (T313559)]], [[gerrit:829043|Disable sticky header edit experiment for idwiki, viwki (T315264)]] (duration: 05m 44s) [production]
20:49 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:44 <thcipriani@deploy1002> thcipriani and cjming and bwang: Backport for [[gerrit:828616|Remove Vector grid config (T313559)]], [[gerrit:829043|Disable sticky header edit experiment for idwiki, viwki (T315264)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
20:44 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:828616|Remove Vector grid config (T313559)]], [[gerrit:829043|Disable sticky header edit experiment for idwiki, viwki (T315264)]] [production]
20:41 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:824787|cirrus: Handle transition to elasticsearch 7.10]] (duration: 16m 56s) [production]
20:40 <ryankemper> T300943 New hosts are in service and were pooled like so: `sudo confctl select name=elastic20[73-86].* set/weight=10:pooled=yes` (in retrospect that syntax seems to have selected too many hosts, but the final state of pybal is correct per https://config-master.wikimedia.org/pybal/codfw/search) [production]
20:39 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1203.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:39 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1202.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:37 <ryankemper@puppetmaster1001> conftool action : set/weight=10:pooled=yes; selector: name=elastic20[73-86].* [production]
20:35 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: T300943 [production]
20:35 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 14 hosts with reason: T300943 [production]
20:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:24 <thcipriani@deploy1002> thcipriani and ebernhardson: Backport for [[gerrit:824787|cirrus: Handle transition to elasticsearch 7.10]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
20:24 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:824787|cirrus: Handle transition to elasticsearch 7.10]] [production]
20:20 <thcipriani@deploy1002> backport aborted: (duration: 03m 09s) [production]
20:20 <thcipriani@deploy1002> backport aborted: (duration: 02m 57s) [production]
20:20 <thcipriani@deploy1002> sync-world aborted: Backport for [[gerrit:829061|Revert "Deploy Research Incentive Survey to idwiki"]] (duration: 01m 23s) [production]
20:20 <thcipriani@deploy1002> thcipriani and trainbranchbot: Backport for [[gerrit:829061|Revert "Deploy Research Incentive Survey to idwiki"]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
20:19 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:829061|Revert "Deploy Research Incentive Survey to idwiki"]] [production]
20:14 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db1203.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:14 <pt1979@cumin1001> START - Cookbook sre.hosts.provision for host db1202.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:13 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1201.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:13 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1200.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:13 <thcipriani@deploy1002> thcipriani and dani: Backport for [[gerrit:828614|Deploy Research Incentive Survey to idwiki (T316466)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
20:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:06 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:828614|Deploy Research Incentive Survey to idwiki (T316466)]] [production]
19:58 <mutante> otrs1001 - sudo systemctl reset-failed - T316903 [production]
19:48 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db1201.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:46 <pt1979@cumin1001> START - Cookbook sre.hosts.provision for host db1200.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1199.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:41 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1198.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:17 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db1199.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:17 <pt1979@cumin1001> START - Cookbook sre.hosts.provision for host db1198.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:16 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1197.mgmt.eqiad.wmnet with reboot policy FORCED [production]