2551-2600 of 10000 results (66ms)
2019-10-31 §
14:50 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1119 at 10%', diff saved to https://phabricator.wikimedia.org/P9512 and previous config saved to /var/cache/conftool/dbconfig/20191031-145010-jynus.json [production]
14:28 <jynus> reloading ferm on db1119 [production]
14:24 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1119', diff saved to https://phabricator.wikimedia.org/P9511 and previous config saved to /var/cache/conftool/dbconfig/20191031-142455-jynus.json [production]
13:40 <effie> upload xdebug 2.7.0-1+wmf2 to component/php72 - T234418 [production]
13:21 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: repool pc1008 T227543 (duration: 01m 02s) [production]
13:16 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1119, db1113 at 10% T227543', diff saved to https://phabricator.wikimedia.org/P9509 and previous config saved to /var/cache/conftool/dbconfig/20191031-131606-jynus.json [production]
11:48 <jynus> setting pc1008 as a replica of active pc1010 [production]
11:43 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: depooling pc1008 T227543 (duration: 01m 01s) [production]
11:37 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1119, db1113 T227543', diff saved to https://phabricator.wikimedia.org/P9507 and previous config saved to /var/cache/conftool/dbconfig/20191031-113659-jynus.json [production]
11:24 <Urbanecm> EU SWAT done [production]
11:23 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.4/extensions/ProofreadPage/: SWAT: e0d5ce9: Add page navigation tabs in correct order skin-side and remove js requirement for Vector tab icons (T231250); ed17da2: Makes sure that Vector default background does not override the navigation arrows (T236969) (duration: 01m 02s) [production]
11:07 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|547086|Enable ContentTranslation out of Beta in Albanian WP (T236064)]] (duration: 01m 02s) [production]
11:03 <ema> cp5008: restart ats-be to clear "backend process restarted" alert [production]
11:00 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:59 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:59 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:59 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:59 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:54 <godog> bounce logstash on logstash2004 [production]
10:39 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:38 <ema> pool cp5009 with ATS backend T227432 [production]
10:37 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:35 <oblivian@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
10:30 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:29 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:19 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:18 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:13 <godog> bounce logstash on logstash2004 [production]
10:07 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
10:05 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
09:46 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:43 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:37 <godog> temporarily stop logstash on logstash2005 to test performance with two ingesters only - T215904 [production]
09:23 <godog> temporarily stop logstash on logstash2006 to test performance with two ingesters only - T215904 [production]
09:10 <ema> depool cp5009 and reimage as text_ats T227432 [production]
08:25 <ariel@deploy1001> Finished deploy [dumps/dumps@f2b6d78]: couple of fixup scripts, bug fix for incr dumps index.html generation (duration: 00m 03s) [production]
08:25 <ariel@deploy1001> Started deploy [dumps/dumps@f2b6d78]: couple of fixup scripts, bug fix for incr dumps index.html generation [production]
06:37 <elukey> upgrade cergen to 0.2.5 on puppetmaster1001 [production]
03:44 <vgutierrez> switch from nginx to ats-tls on cp4032 - T231627 [production]
03:09 <vgutierrez> switch from nginx to ats-tls on cp4031 - T231627 [production]
02:51 <vgutierrez> switch from nginx to ats-tls on cp4030 - T231627 [production]
01:41 <eileen> civicrm revision changed from 0547c84f73 to d2045c6b98, config revision is 1a709a61aa (looks like patch was still hung in gerrit last time) [production]
01:34 <eileen> civicrm revision is 0547c84f73, config revision is 1a709a61aa - that should stop those failmails [production]
00:40 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.4/extensions/WikiLove/resources/ext.wikiLove.icon.vector.css: T236958 Fix Vector icon after upstream change (duration: 01m 02s) [production]
00:38 <eileen> civicrm revision changed from a55c2d2787 to 0547c84f73, config revision is 1a709a61aa [production]
2019-10-30 §
23:21 <ejegg> updated fundraising python tools from ffc7bf764b to a93eec292d [production]
23:08 <XioNoX> power cycle cr3-esams re1 - T236598 [production]
22:29 <mutante> scandium - live hack /srv/mediawiki/wmf-config/InitialiseSettings.php - set wmgMemoryLimit to 850 (*1024 *1024), restart php7.2-fpm (T236833) [production]
22:22 <andrew@deploy1001> Finished deploy [horizon/deploy@2d551d8]: Rolling out a currently-turned-off puppet edit mode (duration: 03m 15s) [production]
22:19 <andrew@deploy1001> Started deploy [horizon/deploy@2d551d8]: Rolling out a currently-turned-off puppet edit mode [production]