2010-06-07
§
|
16:43 |
<Tim> |
moved mediawiki-cvs archive files away to prevent further archiving |
[production] |
16:26 |
<Tim> |
disabled archiving on problematic list mediawiki-cvs |
[production] |
14:07 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
14:04 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
13:49 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
13:48 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
13:46 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
07:43 |
<Tim> |
on lily: fixed broken sources.list and upgraded python, installed python symbols |
[production] |
07:25 |
<Tim> |
on lily: restarted mailman (including ArchRunner) |
[production] |
07:07 |
<Tim> |
on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces |
[production] |
03:41 |
<Tim> |
srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA. |
[production] |
03:25 |
<Fred> |
setup ganglia for lily. |
[production] |
03:07 |
<Tim> |
srv281 down for 41 hours, trying reboot |
[production] |
02:54 |
<Fred> |
upgraded gmond on lily. Modified gmond.conf to handle new modular arch. |
[production] |
02:46 |
<Tim> |
on lily: experimentally disabling broken bayes feature in spamassassin |
[production] |
02:35 |
<Tim> |
on lily: restarting spamd and fixing lock file errors |
[production] |
01:25 |
<Tim> |
on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage |
[production] |
2010-06-06
§
|
11:33 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r67456' |
[production] |
11:33 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r67456' |
[production] |
07:33 |
<Tim> |
on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown |
[production] |
03:10 |
<Tim> |
serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains. |
[production] |
02:14 |
<Tim> |
installed a redirect from en.m.wikipedia.com to .org |
[production] |
01:18 |
<Tim> |
changed the mobile1 apache access log format to something more useful (and squid-like) |
[production] |
00:47 |
<Tim> |
on mobile1: installed logrotate script for apache2 (via puppet) |
[production] |
2010-06-04
§
|
22:52 |
<tomaszf> |
starting webstats with new binary |
[production] |
22:50 |
<tomaszf> |
stopping webstats in prep for update to track mobile stats |
[production] |
19:30 |
<atglenn> |
moved bad snapshots (apr 11 through may 6 2010) to /mnt/dumps/public/bad so public index shows only good dumps and so there will be no prefetch against them |
[production] |
18:47 |
<Fred> |
moved mobile2 to squid vlan / re-ip'ed / dns changed. mobile1 => 115 mobile2 => 116 |
[production] |
18:35 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix. Gotta kill this thing some time' |
[production] |
18:35 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67355' |
[production] |
18:34 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r67355' |
[production] |
12:11 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'WikimediaMobile' |
[production] |
11:37 |
<Tim> |
mobile down for 15 minutes, possibly apache threads exhausted, restarting apache |
[production] |
09:56 |
<catrope> |
synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67333' |
[production] |
09:56 |
<domas> |
deployments manage to kill apache processes sometimes |
[production] |
09:50 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 66620 |
[production] |
09:50 |
<Tim> |
pushing out WikimediaMobile (r67331) in preparation for deployment on testwiki |
[production] |
08:44 |
<domas> |
decreased keepalivetimeout and timeout on mobile1 |
[production] |
08:35 |
<Tim> |
on mobile1: reduced max passenger pool size to 200, Domas and I think it's about right, shouldn't exceed allowable memory, should give us close to 100% CPU. |
[production] |
08:26 |
<Tim> |
on mobile1: domas fixed file limit, now 50k |
[production] |
08:10 |
<Tim> |
increasing MaxClients on mobile1 to 1500 |
[production] |
05:01 |
<Fred> |
Added apache2.conf, memcached.conf to puppet receipe for mobile. |
[production] |
03:43 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '23784 - Modify add/remove rights for bureaucrats on officewiki' |
[production] |
02:46 |
<Tim> |
mobile1: increased ServerLimit to 1500 and reduced MaxClients to 500 |
[production] |
02:35 |
<Tim> |
on mobile1: increased memcached memory limit from 64M to 5000M |
[production] |