251-300 of 7332 results (6ms)
2010-06-06 §
11:33 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r67456' [production]
07:33 <Tim> on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown [production]
03:10 <Tim> serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains. [production]
02:14 <Tim> installed a redirect from en.m.wikipedia.com to .org [production]
01:18 <Tim> changed the mobile1 apache access log format to something more useful (and squid-like) [production]
00:47 <Tim> on mobile1: installed logrotate script for apache2 (via puppet) [production]
2010-06-05 §
21:21 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup' [production]
21:15 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php [production]
20:55 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup' [production]
03:29 <Tim> fixed mobile1 in nagios, added mobile2 [production]
02:09 <Tim> restarting gmond on all miscellaneous cluster servers, to make mobile2 reappear in ganglia (broken by IP renumber) [production]
2010-06-04 §
22:52 <tomaszf> starting webstats with new binary [production]
22:50 <tomaszf> stopping webstats in prep for update to track mobile stats [production]
19:30 <atglenn> moved bad snapshots (apr 11 through may 6 2010) to /mnt/dumps/public/bad so public index shows only good dumps and so there will be no prefetch against them [production]
18:47 <Fred> moved mobile2 to squid vlan / re-ip'ed / dns changed. mobile1 => 115 mobile2 => 116 [production]
18:35 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix. Gotta kill this thing some time' [production]
18:35 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67355' [production]
18:34 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r67355' [production]
12:11 <tstarling> synchronized php-1.5/wmf-config/InitialiseSettings.php 'WikimediaMobile' [production]
11:37 <Tim> mobile down for 15 minutes, possibly apache threads exhausted, restarting apache [production]
09:56 <catrope> synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67333' [production]
09:56 <domas> deployments manage to kill apache processes sometimes [production]
09:50 <tstarling> synchronizing Wikimedia installation... Revision: 66620 [production]
09:50 <Tim> pushing out WikimediaMobile (r67331) in preparation for deployment on testwiki [production]
08:44 <domas> decreased keepalivetimeout and timeout on mobile1 [production]
08:35 <Tim> on mobile1: reduced max passenger pool size to 200, Domas and I think it's about right, shouldn't exceed allowable memory, should give us close to 100% CPU. [production]
08:26 <Tim> on mobile1: domas fixed file limit, now 50k [production]
08:10 <Tim> increasing MaxClients on mobile1 to 1500 [production]
05:01 <Fred> Added apache2.conf, memcached.conf to puppet receipe for mobile. [production]
03:43 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '23784 - Modify add/remove rights for bureaucrats on officewiki' [production]
02:46 <Tim> mobile1: increased ServerLimit to 1500 and reduced MaxClients to 500 [production]
02:35 <Tim> on mobile1: increased memcached memory limit from 64M to 5000M [production]
02:15 <Tim> switched mobile1 over from apache2-mpm-worker to apache2-mpm-prefork (via puppet) [production]
01:03 <Tim> set ganglia host_dmax to 1 day [production]
2010-06-03 §
21:57 <Fred> mobile1 re-imaged and puppetized. Changed subnet for mobile1. Changed DNS for mobile1. m pointing to newly imaged mobile1 (until transition is completed) [production]
20:55 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '23689 - Enable Collection extension on Thai Wikipedia' [production]
20:22 <AaronSchulz> deployed r67296 FlaggedRevs_alpha [production]
20:21 <aaron> synchronizing Wikimedia installation... Revision: 66620 [production]
19:39 <mark> Moved mobile1 switchport from vlan 101 to 100 [production]
19:36 <mark> Reverted DNS change of mobile1, back to .157 [production]
17:21 <Fred> mobile1 going to be unreacheable while re-ip'ing [production]
14:05 <midom> synchronized php-1.5/wmf-config/InitialiseSettings.php 'timezone change for bat-smg' [production]
11:53 <mark> Made m.wikipedia.org CNAME m.wikipedia.org, m.wikipedia.org A to mobile1/2 in RR [production]
10:57 <hcatlin> mobile2 has been rebuilt and is featuring the new apache/mobile stack taking 40% of all mobile traffic. pls help monitor on ganglia. [production]
09:04 <Tim> cleaning COSS on sq45, resynced its configuration, will start squid when done [production]
08:58 <Tim> kernel reports degraded RAID on sq33, sq34, sq35, sq37, sq38, sq40 [production]
08:39 <Tim> checked all serial consoles, all nonresponsive, rebooted all [production]
08:23 <Tim> sq33, sq34, sq35, sq37, sq38, sq40, sq45 have been down for 16-28 days, apparently for no good reason, can't find any log or DT entries. Will try restarts. [production]
07:56 <Tim> added new squids to nagios [production]
06:36 <Tim> cleaning cache directories on sq56 to avoid resurrection of expired content [production]