651-700 of 10000 results (58ms)
2019-12-02 ยง
17:14 <ssastry@deploy1001> Started deploy [parsoid/deploy@743efb0]: Updating Parsoid to ca588b25 + fix broken langconv library / deploy [production]
17:09 <ejegg> disabled fundraising job omnimail_groupmember_load [production]
16:45 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:43 <ejegg> updated fundraising internal dashboard from 8fc2726736 to 3a93d2aba4 [production]
16:43 <effie> restart all API cluster in eqiad [production]
16:43 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:42 <hashar> Restarted CI Jenkins [production]
16:41 <mobrovac@deploy1001> Finished deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015 (duration: 13m 53s) [production]
16:41 <ema> cp3050: ats-be restart with proxy.config.http.server_session_sharing.pool=global T238494 [production]
16:32 <ema> cp3053: repooling after firmware update T239041 [production]
16:27 <mobrovac@deploy1001> Started deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015 [production]
16:19 <effie> reimage mw1295.eqiad.wmnet mw1294.eqiad.wmnet mw1293.eqiad.wmnet [production]
16:11 <robh> cp3053 depooling and rebooting for firmware update T239041 [production]
16:10 <robh> cp3035 depooling and rebooting for firmware update T239041 [production]
15:38 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
15:38 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Parsoid VRS: Switch groups 0 and 1 to Parsoid/PHP - T229015 (duration: 00m 59s) [production]
15:35 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
15:30 <mobrovac@deploy1001> Finished deploy [restbase/deploy@d6d5a6e]: Parsoid Proxy: Do not use the fall-back for linting transforms - T239607 (duration: 14m 51s) [production]
15:26 <effie> Rolling restart mw1345-1348 [production]
15:15 <mobrovac@deploy1001> Started deploy [restbase/deploy@d6d5a6e]: Parsoid Proxy: Do not use the fall-back for linting transforms - T239607 [production]
14:46 <ema> cp-ats: set server_session_sharing.match=2 everywhere (puppet re-enable and run) T238494 [production]
14:31 <ema> cp-ats: merge server_session_sharing.match=2 (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/553490/) with puppet disabled, test on cp3050 T238494 [production]
14:18 <godog> set grafana theme back to light, was dark for some reason [production]
14:10 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:08 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P9794 and previous config saved to /var/cache/conftool/dbconfig/20191202-135643-marostegui.json [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1075 after schema change', diff saved to https://phabricator.wikimedia.org/P9793 and previous config saved to /var/cache/conftool/dbconfig/20191202-135543-marostegui.json [production]
13:47 <ema> power-cycle cp3053 T239041 [production]
13:44 <hashar> Restarted CI Jenkins [production]
13:30 <hashar> Restarted CI Jenkins [production]
13:14 <mobrovac@deploy1001> Finished deploy [restbase/deploy@eedba38]: Parsoid Proxy: Fixes - T229015 (duration: 14m 49s) [production]
13:04 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:01 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:59 <mobrovac@deploy1001> Started deploy [restbase/deploy@eedba38]: Parsoid Proxy: Fixes - T229015 [production]
12:57 <mobrovac@deploy1001> Finished deploy [restbase/deploy@eedba38] (dev-cluster): Parsoid Proxy: Fixes (duration: 02m 54s) [production]
12:54 <mobrovac@deploy1001> Started deploy [restbase/deploy@eedba38] (dev-cluster): Parsoid Proxy: Fixes [production]
12:54 <Urbanecm> EU SWAT done [production]
12:50 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: d27fe78: Enable partial blocks on eswiki (T239370) (duration: 01m 00s) [production]
12:45 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 445bdc3: Remove `move-rootuserpages` from user on svwiki (T238842) (duration: 01m 04s) [production]
12:43 <Urbanecm> Purge https://en.wikipedia.org/static/images/project-logos/bawiki*.png [production]
12:39 <urbanecm@deploy1001> Synchronized static/images/project-logos/: SWAT: 61a9563: Revert "Change bawiki logo to an anniversary one" (T237070) (duration: 01m 06s) [production]
12:37 <effie> reimage mw1296.eqiad.wmnet [production]
12:37 <effie> reimage mw1298.eqiad.wmnet [production]
12:31 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:554049|Set read new for term store for items of wikidata up to Q1000 (T225057)]] (duration: 01m 00s) [production]
12:19 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.8/extensions/GrowthExperiments/: SWAT: [[gerrit:553402|Suggested edits: do not treat AQS lookup failure as error (T238178)]] (duration: 01m 02s) [production]
11:58 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:56 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:50 <filippo@cumin1001> conftool action : set/pooled=yes; selector: name=mw2229.codfw.wmnet [production]
11:38 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:554033| Bumping portals to master (T128546)]] (duration: 01m 00s) [production]
11:37 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:554033| Bumping portals to master (T128546)]] (duration: 01m 04s) [production]