551-600 of 10000 results (15ms)
2014-01-17 §
09:44 <springle> synchronized wmf-config/db-eqiad.php 'LB 'dump' s[2-7]' [production]
08:50 <manybubbles> finished elasticsearch upgrade. we're now 0.90.10 all the way. [production]
08:50 <ori> Replacing StatsD on tungsten with txStatsD; see commit I19ecf608d for rationale. [production]
08:50 <ori> restarted morebots. morebots missed Sean's 'repool db1007, depool db1041' @ 6:26 UTC. [production]
2014-01-16 §
19:22 <reedy> synchronized database lists files: [production]
19:20 <Reedy> Created zhwikivoyage echo tables on db1029 [production]
19:18 <LeslieCarr> reseating pem0 on cr2-ulsfo to try and clear problem [production]
19:13 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, testwiki, testwikidatawiki and mediawikiwiki to 1.23wmf11 [production]
19:12 <^d> dropped all cirrusSearchLinksUpdate jobs from enwiki job queue. Job queue back to ~330k entries, far better than the ~3mil [production]
19:06 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.23wmf10 [production]
18:50 <paravoid> powercycling ms-be1002, down, console unresponsive [production]
18:44 <manybubbles> starting Elasticsearch upgrade from 0.90.9 to 0.90.10 [production]
18:01 <reedy> synchronized php-1.23wmf11/includes/specials/SpecialWatchlist.php [production]
17:54 <reedy> synchronized docroot and w [production]
17:45 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf9 till window [production]
17:29 <reedy> finished scap: testwiki to 1.23wmf11 and build l10n cache (duration: 23m 24s) [production]
17:08 <reedy> started scap: testwiki to 1.23wmf11 and build l10n cache [production]
16:48 <cmjohnson1> es1007 replacing disk slot 6 [production]
16:47 <cmjohnson1> tungsten replacing failing disk slot 3 [production]
16:47 <cmjohnson1> db1004 replacing failing disk slot 9 [production]
16:43 <reedy> synchronized php-1.23wmf11 [production]
16:14 <bblack> rebooting cp1055 to clear out mess from XFS/kmem_alloc bug [production]
15:48 <cmjohnson1> disabling puppet on db29 [production]
11:55 <akosiaris> fixed the cp1065 puppet freshness check constantly bugging us and reenabled notifications after that [production]
09:19 <ori> updated /a/common to {{Gerrit|Id13e614e5}}: repool db1042 [production]
06:01 <springle> synchronized wmf-config/db-eqiad.php 'repool db1042' [production]
05:58 <springle> synchronized wmf-config/db-eqiad.php 'depool db1033' [production]
05:21 <springle> synchronized wmf-config/db-eqiad.php 'depool db1042' [production]
03:29 <LocalisationUpdate> ResourceLoader cache refresh completed at 2014-01-16 03:28:58+00:00 [production]
03:00 <Coren> Deleted all non-current pacct on neon; activity seems to have increased tenfold and /var/log is OOS [production]
02:51 <LocalisationUpdate> completed (1.23wmf9) at 2014-01-16 02:51:29+00:00 [production]
02:27 <LocalisationUpdate> completed (1.23wmf10) at 2014-01-16 02:27:00+00:00 [production]
01:30 <catrope> synchronized php-1.23wmf10/extensions/VisualEditor 'Forgot to run git submodule update earlier' [production]
01:28 <catrope> synchronized php-1.23wmf10/resources/startup.js 'touch' [production]
01:27 <catrope> synchronized php-1.23wmf10/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' [production]
01:27 <catrope> synchronized php-1.23wmf9/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' [production]
01:00 <aaron> synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php '1218cc7e9cba8342d0184523f1d7c1fec608e656' [production]
00:49 <demon> synchronized wmf-config/InitialiseSettings.php 'Config new Cirrus logs' [production]
00:43 <catrope> synchronized php-1.23wmf10/extensions/VisualEditor 'Update VE for cherry-picks' [production]
00:43 <catrope> synchronized php-1.23wmf9/extensions/VisualEditor 'Update VE for cherry-picks' [production]
00:29 <mflaschen> synchronized php-1.23wmf10/extensions/GettingStarted/ 'Sync GettingStarted wmf10 for hotfix' [production]
00:29 <mflaschen> synchronized php-1.23wmf9/extensions/GettingStarted/ 'Sync GettingStarted wmf9 for hotfix' [production]
00:16 <mflaschen> synchronized wmf-config/InitialiseSettings.php 'Deploy GuidedTour to astwiki, fawiki, and ruwiki' [production]
00:07 <Tim> rebooting cp1066 following XFS deadlock [production]
2014-01-15 §
23:33 <Tim> on cp1066: stopping varnish -- kswapd has gone crazy and is continuously flushing all buffers [production]
21:31 <hashar> Jenkins: generation of MediaWiki documentation with doxygen got broken since Zuul upgrade. Fix: {{gerrit|107718}} [production]
21:17 <aaron> synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php 'bc2a0ddbff4c3e4bff8e931163fd3d1c0340a78e' [production]
20:50 <^d> gerrit: running gc on all repositories. think weekly cron to do this might be broken. [production]
20:49 <manybubbles> deployment-prep updating elasticsearch to 0.90.10 [production]
20:26 <cmjohnson1> db1031 swapping failing disk slot 11 [production]