9251-9300 of 10000 results (33ms)
2014-03-11 §
02:12 <LocalisationUpdate> completed (1.23wmf16) at 2014-03-11 02:12:35+00:00 [production]
2014-03-10 §
23:30 <^d> kicking gerrit to pick up bugfix. [production]
23:07 <mutante> shutting down srv258-srv270 [production]
22:18 <bd808> Two instances of logstash were running on logstash1001; Killed both and started service again [production]
21:55 <bd808> Restarted logstash on logstash1001; new events flowing in again now [production]
21:49 <K4-713> synchronized payments to 5d20972. [production]
21:47 <bd808> ganglia monitoring for elasticsearch on logstash cluster seems broken. Caused by 1.0.x upgrade having not happened there yet? [production]
21:46 <bd808> Restarted elastcisearch on logstash1003; it was JVM heap thrashing at 98% heap used. [production]
21:44 <RobH> arsenic reclaim per rt6522, ignore alerts [production]
21:40 <sbernardin> ms-be5 swapping failed disk [production]
21:34 <K4-713> synchronized payments cluster to 01f7af8 [production]
21:27 <bd808> No new data in logstash since 14:56Z. Bryan will investigate. [production]
20:26 <gwicke> Coren fixed up the Parsoid deploy by running "salt-run deploy.restart 'parsoid/deploy' '10%'" from the salt master as a work-around for [[bugzilla:61882]] [production]
20:18 <gwicke> deployed Parsoid 681f7b8d2 using deploy 77d17489; service restart incomplete due to [[bugzilla:61882]] [production]
19:44 <reedy> synchronized wmf-config/ 'Use Vips for images over 20MP' [production]
19:17 <^d> enwiki search indexes don't look happy. sporadic reports of "search request timed out" from users, cpu usage pretty high. possibly unrelated bug logs getting spammed with: "Retry, temporal error for query..." [production]
18:50 <awight> updated crm from b85722a7c14af6ec1b50d2cddcd9f84d8c88da3e to a1ca31664f771a48963425893b2b927a5148119b [production]
18:27 <awight> crm updated from 1aa34fd565e818466515b70c6a00af2e30e47ce2 to b85722a7c14af6ec1b50d2cddcd9f84d8c88da3e [production]
18:11 <krinkle> synchronized php-1.23wmf17/resources/mediawiki.api/mediawiki.api.watch.js 'I2ac9e0da0f1c825' [production]
18:07 <demon> synchronized php-1.23wmf17/extensions/Elastica/Elastica.php 'I forgot how to use git' [production]
18:03 <demon> synchronized php-1.23wmf17/extensions/Elastica/Elastica.php [production]
17:37 <andrewbogott> upgrading mediawiki on wikitech [production]
16:33 <hashar> restarted Zuul on gallium : leaked file descriptors (fixed upstream) [production]
15:43 <hoo> synchronized multiversion/MWScript.php 'Path typo fix: {{Gerrit|I6a05447}}' [production]
15:06 <hoo> synchronized wmf-config/session-labs.php 'Syncing labs file for consistency {{Gerrit|I5e1a3242}}' [production]
15:05 <hoo> synchronized wmf-config/InitialiseSettings.php 'Enable Extension:GuidedTour on testwikidatawiki' [production]
14:58 <reedy> synchronized wmf-config/InitialiseSettings.php [production]
14:57 <reedy> updated /a/common to {{Gerrit|Ibcbd10044}}: Disable and remove ContactPageFundraiser [production]
14:44 <reedy> synchronized wmf-config/ 'Remove ContactPageFundraiser, SkinPerPage, FormPreloadPostCache and skins used on foundationwiki' [production]
14:26 <ottomata> put elastic1007 and elastic1013-1016 back in main elasticsearch pool [production]
09:11 <hashar> Jenkins restarted and proceeding jobs again [production]
09:06 <hashar> stopping Jenkins [production]
09:01 <hashar> Jenkins unresponsive for some reason yeah!!! [production]
02:57 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Mar 10 02:57:32 UTC 2014 (duration 57m 31s) [production]
02:25 <LocalisationUpdate> completed (1.23wmf17) at 2014-03-10 02:25:14+00:00 [production]
02:14 <LocalisationUpdate> completed (1.23wmf16) at 2014-03-10 02:14:08+00:00 [production]
2014-03-09 §
19:25 <awight> crm updated from f06bd7920921205e97a142cec19f1096b9761768 to 1aa34fd565e818466515b70c6a00af2e30e47ce2 [production]
2014-03-08 §
03:49 <mutante> shut down srv250-srv247 (this was stdpa row D-D2) [production]
03:42 <mutante> shutting down srv193,srv235-srv247,srv248,249 [production]
03:28 <andrewbogott> upgrading nova-compute on virt100[1-6] [production]
03:15 <mutante> shut down Tampa mw hosts 21-27,49,58,75-85 (that is sdpta row D:D1) [production]
03:04 <mutante> shut down Tampa mw hosts 1-20 [production]
02:56 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Mar 8 02:55:55 UTC 2014 (duration 55m 54s) [production]
02:25 <LocalisationUpdate> completed (1.23wmf17) at 2014-03-08 02:25:00+00:00 [production]
02:14 <LocalisationUpdate> completed (1.23wmf16) at 2014-03-08 02:14:25+00:00 [production]
2014-03-07 §
23:36 <ori> updated /a/common to {{Gerrit|If2530281b}}: Order branch directories as version numbers [production]
22:14 <mutante> deleting salt keys for all Tampa app servers removed from puppet [production]
21:54 <hashar> Jenkins restarted [production]
21:49 <hashar> restarting Jenkins it is broken [production]
21:45 <mutante> killing all tampa appservers from puppetstoredconfigs [production]