1701-1750 of 10000 results (26ms)
2015-02-10 §
00:13 <bd808> restarted elasticsearch on logstash1001; OOM [production]
2015-02-09 §
23:55 <yurik> Started scap: syncing ZeroBanner i18n [production]
23:54 <yurik> Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 05s) [production]
23:54 <yurik> Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189617 (duration: 00m 07s) [production]
23:01 <yurik> Synchronized php-1.25wmf16/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s) [production]
23:00 <yurik> Synchronized php-1.25wmf15/extensions/ZeroBanner: cherry-picking 189553 (duration: 00m 06s) [production]
17:46 <bd808> restarted elasticsearch on logstash1003; OOM [production]
16:53 <marktraceur> Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s) [production]
16:48 <marktraceur> Synchronized php-1.25wmf16/extensions/UploadWizard/: [SWAT] [wmf16] Trying to force UploadWizard to update (duration: 00m 05s) [production]
16:48 <marktraceur> Synchronized php-1.25wmf15/extensions/UploadWizard/: [SWAT] [wmf15] Trying to force UploadWizard to update (duration: 00m 06s) [production]
16:47 <jgage> restarted eventlogging on hafnium (with deploy from master on tin this time) [production]
16:38 <marktraceur> Synchronized php-1.25wmf16/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf16] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 06s) [production]
16:37 <marktraceur> Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FlickrChecker.js: [SWAT] [wmf15] Re-add flickrreview template to files imported from Flickr by UploadWizard (duration: 00m 05s) [production]
16:22 <jgage> restarted eventlogging on hafnium for nuria via ~root/upgrade-eventlogging --no-update [production]
16:18 <marktraceur> Synchronized wmf-config/throttle.php: [SWAT] [config] Add throttle rules for two workshops (duration: 00m 07s) [production]
16:14 <marktraceur> Synchronized wmf-config/: [SWAT] [config] Un-subscribe frequently failing recipients (duration: 00m 05s) [production]
16:12 <marktraceur> Synchronized php-1.25wmf16/extensions/OAuth/: [SWAT] [wmf16] OAuth: Support ListDefinedTags and ChangeTagsListActive hooks (duration: 00m 11s) [production]
15:00 <cmjohnson1> cp1070 down for h/w troubleshooting. Already depooled by bblack [production]
11:58 <godog> bounce mwprof-profiler-to-carbon on tungsten [production]
10:47 <hoo> Manually removed wikidatawiki.wb_changes_dispatch entries for test wikis (test2wiki, testwiki, testwikidata). [production]
09:11 <gwicke> cassandra load testing on xenon, praseodymium and cerium; disk space is tight, might run out on one of those boxes but they are purely test boxes right now, so np [production]
05:32 <gwicke> stopped puppet on cerium, praseodymium & xenon [production]
05:31 <gwicke> manually updated cassandra on cerium, praseodymium & xenon to 2.1.2 (see https://phabricator.wikimedia.org/T88956) [production]
03:50 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Feb 9 03:48:57 UTC 2015 (duration 48m 56s) [production]
02:13 <LocalisationUpdate> completed (1.25wmf16) at 2015-02-09 02:12:45+00:00 [production]
02:12 <l10nupdate> Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s) [production]
02:12 <LocalisationUpdate> completed (1.25wmf15) at 2015-02-09 02:11:13+00:00 [production]
02:11 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
2015-02-08 §
03:53 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Feb 8 03:52:49 UTC 2015 (duration 52m 48s) [production]
02:14 <LocalisationUpdate> completed (1.25wmf16) at 2015-02-08 02:13:11+00:00 [production]
02:13 <l10nupdate> Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s) [production]
02:12 <LocalisationUpdate> completed (1.25wmf15) at 2015-02-08 02:11:41+00:00 [production]
02:11 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
2015-02-07 §
15:36 <apergos> started nginx on daaset1001, it was not running for some reason [production]
09:40 <bblack> depooled cp1070 in pybal [production]
09:33 <bblack> rebooting cp1070 (dead network, dead console) [production]
05:10 <subbu> deployed parsoid hotfiix 8ca7ef40 (cherry-pick of 447a0565) [production]
04:48 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Feb 7 04:47:30 UTC 2015 (duration 47m 29s) [production]
03:13 <gwicke> restarting parsoid cluster [production]
02:34 <LocalisationUpdate> completed (1.25wmf16) at 2015-02-07 02:33:09+00:00 [production]
02:33 <l10nupdate> Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 01s) [production]
02:20 <LocalisationUpdate> completed (1.25wmf15) at 2015-02-07 02:19:09+00:00 [production]
02:19 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s) [production]
02:11 <qchris> Ran kafka leader re-election as analytics1021 dropped out of it's partition leader role. [production]
01:48 <bblack> leaving cp1064 (jessie upload eqiad) pooled front+back. it's experimental but looks stable. if upload-related 503 spikes and I'm not around, feel free to depool it. [production]
00:18 <qchris> Manually bumping heap for the Hadoop namenodes and revived them after both of them running out of heap and not coming back. [production]
2015-02-06 §
22:53 <marktraceur> Synchronized wmf-config/: [friday] beta config change for tgr (duration: 00m 09s) [production]
22:53 <subbu> restarted parsoid service to kill several stuck processes on multiple nodes [production]
20:05 <robh> ms1004 coming offline, shouldnt page (but disregard if it does) [production]
19:19 <subbu> deployed parsoid hotfiix a9dbd4fc (cherry-pick of 76d6658c) [production]