8301-8350 of 10000 results (96ms)
2022-12-05 ยง
11:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on idp-test1002.wikimedia.org with reason: Various tests which may cause temporary breakage on idp-test.w.o [production]
11:09 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on idp-test1002.wikimedia.org with reason: Various tests which may cause temporary breakage on idp-test.w.o [production]
11:07 <hashar> Restarted Zuul to clear a stuck ssh connection with Gerrit - T309376 [production]
10:33 <kostajh> UTC morning deploys done [production]
10:32 <godog> contint1001 - racadm serveraction powercyle - crashed [production]
10:31 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864713|User impact: Show discovery notice to mobile users (T323619)]] (duration: 09m 30s) [production]
10:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42241 and previous config saved to /var/cache/conftool/dbconfig/20221205-103028-marostegui.json [production]
10:23 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864713|User impact: Show discovery notice to mobile users (T323619)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
10:22 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864713|User impact: Show discovery notice to mobile users (T323619)]] [production]
10:14 <Emperor> rebalance thanos rings T311690 [production]
10:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42240 and previous config saved to /var/cache/conftool/dbconfig/20221205-100607-marostegui.json [production]
10:05 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864712|User impact: Show discovery tour to desktop users who had old module (T323619)]] (duration: 27m 33s) [production]
09:50 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864712|User impact: Show discovery tour to desktop users who had old module (T323619)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
09:39 <moritzm> restarting mediawiki canaries to pick up freetype security updates [production]
09:38 <godog> force a puppet run on physical hosts to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/860572 [production]
09:37 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864712|User impact: Show discovery tour to desktop users who had old module (T323619)]] [production]
09:36 <moritzm> installing freetype security updates [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42239 and previous config saved to /var/cache/conftool/dbconfig/20221205-091547-marostegui.json [production]
09:15 <kharlan@deploy1002> backport aborted: (duration: 00m 25s) [production]
09:14 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:864666|Fix ExpensiveUserImpact input validation (T324312)]] (duration: 09m 10s) [production]
09:06 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:864666|Fix ExpensiveUserImpact input validation (T324312)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
09:05 <kharlan@deploy1002> Started scap: Backport for [[gerrit:864666|Fix ExpensiveUserImpact input validation (T324312)]] [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42238 and previous config saved to /var/cache/conftool/dbconfig/20221205-090214-marostegui.json [production]
09:00 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 59689 [production]
09:00 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 59689 [production]
09:00 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 58308 [production]
08:58 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 58308 [production]
08:58 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 141731 [production]
08:55 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 141731 [production]
08:54 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52580 [production]
08:53 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 52580 [production]
08:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42237 and previous config saved to /var/cache/conftool/dbconfig/20221205-085235-marostegui.json [production]
08:52 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 136907 [production]
08:51 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 136907 [production]
08:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 55818 [production]
08:49 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 55818 [production]
08:48 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 38623 [production]
08:48 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:859991|GrowthExperiments: End imagerecommendation experiment (T323686)]] (duration: 09m 26s) [production]
08:47 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 38623 [production]
08:43 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 4788 [production]
08:40 <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:859991|GrowthExperiments: End imagerecommendation experiment (T323686)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
08:38 <kharlan@deploy1002> Started scap: Backport for [[gerrit:859991|GrowthExperiments: End imagerecommendation experiment (T323686)]] [production]
08:38 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 4788 [production]
08:35 <kartik@deploy1002> Finished scap: Backport for [[gerrit:863097|Enable Section Translation on 8 Wikipedias (T319176)]] (duration: 09m 57s) [production]
08:29 <filippo@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=thanos-web,name=eqiad [production]
08:27 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:863097|Enable Section Translation on 8 Wikipedias (T319176)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
08:25 <kartik@deploy1002> Started scap: Backport for [[gerrit:863097|Enable Section Translation on 8 Wikipedias (T319176)]] [production]
08:24 <filippo@cumin1001> conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet,service=thanos-web [production]
08:23 <filippo@cumin1001> conftool action : set/pooled=no; selector: name=thanos-fe2003.codfw.wmnet,service=thanos-web [production]
08:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1206 with more weight', diff saved to https://phabricator.wikimedia.org/P42236 and previous config saved to /var/cache/conftool/dbconfig/20221205-082320-marostegui.json [production]