301-350 of 7406 results (5ms)
2010-06-07 §
20:55 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'rollback for de_labs editors' [production]
19:46 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up fix for tmp_id' [production]
19:40 <catrope> synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'r67558' [production]
19:12 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up pagination from r67553' [production]
18:56 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'Picking up pagination from r67553' [production]
16:57 <Fred> Limiting message size to 5M for lists. [production]
16:43 <Tim> moved mediawiki-cvs archive files away to prevent further archiving [production]
16:26 <Tim> disabled archiving on problematic list mediawiki-cvs [production]
14:07 <andrew> synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php [production]
14:04 <andrew> synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php [production]
13:49 <andrew> synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php [production]
13:48 <andrew> synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php [production]
13:46 <andrew> synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php [production]
07:43 <Tim> on lily: fixed broken sources.list and upgraded python, installed python symbols [production]
07:25 <Tim> on lily: restarted mailman (including ArchRunner) [production]
07:07 <Tim> on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces [production]
03:41 <Tim> srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA. [production]
03:25 <Fred> setup ganglia for lily. [production]
03:07 <Tim> srv281 down for 41 hours, trying reboot [production]
02:54 <Fred> upgraded gmond on lily. Modified gmond.conf to handle new modular arch. [production]
02:46 <Tim> on lily: experimentally disabling broken bayes feature in spamassassin [production]
02:35 <Tim> on lily: restarting spamd and fixing lock file errors [production]
01:25 <Tim> on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage [production]
2010-06-06 §
11:33 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r67456' [production]
11:33 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r67456' [production]
07:33 <Tim> on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown [production]
03:10 <Tim> serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains. [production]
02:14 <Tim> installed a redirect from en.m.wikipedia.com to .org [production]
01:18 <Tim> changed the mobile1 apache access log format to something more useful (and squid-like) [production]
00:47 <Tim> on mobile1: installed logrotate script for apache2 (via puppet) [production]
2010-06-05 §
21:21 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup' [production]
21:15 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php [production]
20:55 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup' [production]
03:29 <Tim> fixed mobile1 in nagios, added mobile2 [production]
02:09 <Tim> restarting gmond on all miscellaneous cluster servers, to make mobile2 reappear in ganglia (broken by IP renumber) [production]
2010-06-04 §
22:52 <tomaszf> starting webstats with new binary [production]
22:50 <tomaszf> stopping webstats in prep for update to track mobile stats [production]
19:30 <atglenn> moved bad snapshots (apr 11 through may 6 2010) to /mnt/dumps/public/bad so public index shows only good dumps and so there will be no prefetch against them [production]
18:47 <Fred> moved mobile2 to squid vlan / re-ip'ed / dns changed. mobile1 => 115 mobile2 => 116 [production]
18:35 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix. Gotta kill this thing some time' [production]
18:35 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67355' [production]
18:34 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r67355' [production]
12:11 <tstarling> synchronized php-1.5/wmf-config/InitialiseSettings.php 'WikimediaMobile' [production]
11:37 <Tim> mobile down for 15 minutes, possibly apache threads exhausted, restarting apache [production]
09:56 <catrope> synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67333' [production]
09:56 <domas> deployments manage to kill apache processes sometimes [production]
09:50 <tstarling> synchronizing Wikimedia installation... Revision: 66620 [production]
09:50 <Tim> pushing out WikimediaMobile (r67331) in preparation for deployment on testwiki [production]
08:44 <domas> decreased keepalivetimeout and timeout on mobile1 [production]
08:35 <Tim> on mobile1: reduced max passenger pool size to 200, Domas and I think it's about right, shouldn't exceed allowable memory, should give us close to 100% CPU. [production]