951-1000 of 10000 results (27ms)
2015-05-13 §
17:10 <manybub|lunch> elastic1002 restarted and rejoined the cluster - now the cluster is repaining. hurray. [production]
17:08 <manybub|lunch> elastic1001 restarted and rejoined the cluster hapilly while I was at lunch. it looks good - no errors beyond the ones we have fixes in flight for. So I'm going to do elastic1002 [production]
17:03 <hashar> Zuul clone failures solved. Was due to network traffic being interrupted between labs and prod. [production]
16:53 <krenair> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/209967/ (duration: 00m 14s) [production]
16:51 <hashar> Zuul clone failure https://phabricator.wikimedia.org/T98980 [production]
16:49 <andrewbogott> re-enabling puppet on labnet1001 [production]
16:46 <mutante> es2010 failed disk, reopening ticket for last fail in January [production]
16:41 <jynus> Enabling puppet agent in db1009.eqiad after reinstall [production]
16:40 <ori> Synchronized php-1.26wmf4/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 11s) [production]
16:38 <ori> Synchronized php-1.26wmf5/includes/resourceloader/ResourceLoader.php: I30b490e5b: ResourceLoader::filter: use APC when running under HHVM (duration: 00m 14s) [production]
16:28 <andrewbogott> disabling puppet on labnet1001 to tinker with nova config [production]
15:44 <mark> Disregard cr2-knams:xe-0/0/0; we're working on it [production]
15:21 <manybubbles> I think the elasticsearch cluster got stuck with alloation disabled after the rolling restart. Funky. Haven't seen that one before. Probably a problem with our instructions. Anyway, unstuck it and recovery is going faster now [production]
15:17 <demon> Synchronized wmf-config/InitialiseSettings.php: didn't work, undoing previous sync (duration: 00m 12s) [production]
15:15 <demon> Synchronized wmf-config/InitialiseSettings.php: trying something (duration: 00m 12s) [production]
14:53 <manybubbles> elasticsearch restart on elastic1001 going well. cluster still in recovering state as expect. I'll give it an hour to soak. [production]
14:48 <manybubbles> ok - time to start the rolling restart. I'm going to to elastic1001 first non-automated and watch it [production]
14:36 <manybubbles> s/gitfit/gitfat/ oh well [production]
14:35 <manybubbles> first attempt at syncing elasticsearch plugins didn't work 100%. syncing again. gitfit/gitdeploy is betraying me [production]
14:32 <manybubbles> syncing new versions of elsaticsearch plugins to prod. no restarts yet. [production]
14:04 <aude> Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking for Wikisource (duration: 00m 14s) [production]
13:57 <aude> added wbc_entity_usage table on all Wikibase Client wikis [production]
13:56 <jynus> jcrespo Disabling puppet agent in db1009.eqiad in preparation for reinstall [production]
13:45 <aude> Synchronized php-1.26wmf5/extensions/Wikidata: Update maintenance script (duration: 00m 20s) [production]
12:45 <springle> xtrabackup clone db1060 to db1018 [production]
12:39 <springle> upgrade and restart db1060 [production]
09:20 <jamesofur> inserting FDC election encryption key [production]
06:21 <LocalisationUpdate> ResourceLoader cache refresh completed at Wed May 13 06:19:59 UTC 2015 (duration 19m 58s) [production]
05:53 <springle> reinstall db1018 [production]
04:50 <springle> Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 12s) [production]
03:11 <LocalisationUpdate> completed (1.26wmf5) at 2015-05-13 03:10:31+00:00 [production]
03:07 <l10nupdate> Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 43s) [production]
02:46 <LocalisationUpdate> completed (1.26wmf4) at 2015-05-13 02:45:28+00:00 [production]
02:39 <l10nupdate> Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 10m 08s) [production]
01:56 <damagecat> Started 'jobs' screen in tin to drain refreshLinks for enwiki using --nothrottle (T98621) [production]
01:29 <legoktm> Synchronized wmf-config/CommonSettings.php: Hardcode UploadWizard max upload size - T98933 (duration: 00m 12s) [production]
01:23 <legoktm> Synchronized php-1.26wmf5/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s) [production]
01:21 <legoktm> Synchronized php-1.26wmf4/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s) [production]
01:07 <gwicke> added commons to supported projects in RESTBase API [production]
00:16 <ori> Synchronized wmf-config/CommonSettings.php: I5ebedfdfb: Set $wgGadgetsCacheType to CACHE_ACCEL (duration: 00m 12s) [production]
00:13 <ori> Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: 914d71f3cc: Temporary hack to drain excess refreshLinks jobs (duration: 00m 14s) [production]
00:12 <ori> Synchronized php-1.26wmf4/extensions/Gadgets: 7539873979: Update Gadgets for cherry-pick (duration: 00m 12s) [production]
00:11 <ori> Synchronized php-1.26wmf5/extensions/Gadgets: cbb9b1e475: Update Gadgets for cherry-pick (duration: 00m 12s) [production]
2015-05-12 §
23:40 <ori> Upgraded all Apaches to HHVM 3.6.1+dfsg1-1+wm2 and Apache 2.4.7-1ubuntu4.4 [production]
23:26 <demon> Synchronized php-1.26wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 12s) [production]
23:24 <demon> Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 11s) [production]
23:23 <demon> Synchronized php-1.26wmf5/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 12s) [production]
23:23 <demon> Synchronized php-1.26wmf5/includes/media/DjVu.php: (no message) (duration: 00m 12s) [production]
23:18 <ori> Upgrading more HHVMs; DPKG alerts likely but they will be transient. [production]
23:11 <demon> Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 11s) [production]