8001-8050 of 10000 results (32ms)
2011-07-20 §
15:27 <RobH> powercycling srv283 [production]
15:27 <RobH> srv283 is hung up again, and now wont take my commands in serial console... [production]
15:22 <RobH> srv283 loads are insane, its very slugging, and only serial console works. kicking the apache process over to free up memory and resume normal function [production]
14:39 <RobH> srv239 unresponsive, rebooting [production]
14:02 <mark> Setup nas1 with clustering failover [production]
08:44 <mark> Fixed misc::mediawiki-logger manifest [production]
00:04 <Ryan_Lane> test [[bugzilla:1|bug 1]] [production]
2011-07-19 §
23:59 <binasher> also set long_query_time to 0.8 on the enwiki read slaves [production]
23:09 <Ryan_Lane> added new ganglia clusters for ssl cluster in pmtpa and esams [production]
22:50 <binasher> reduced long_query_time on db36 to 800ms [production]
22:46 <binasher> setting up mk-query-digest slow query processing on db36 [production]
22:43 <binasher> installing maatkit on core dbs via puppet [production]
22:13 <Jeff_Green> installing mysql 5.0.51a on db1025 to for mydumper/myloader mysqld version upgrade testing [production]
21:40 <Ryan_Lane> make that changing the CNAME to wikipedia-lb [production]
21:40 <Ryan_Lane> changing DNS entry for test.wikipedia.org to wikipedia-lb.wikimedia.org [production]
21:04 <catrope> synchronized php-1.17/resources/startup.js 'touch' [production]
21:03 <catrope> synchronized wmf-config/CommonSettings.php 'Kill underscores in blacklist category name' [production]
20:58 <catrope> synchronized php-1.17/resources/startup.js 'touch' [production]
20:58 <catrope> synchronized wmf-config/CommonSettings.php 'Set up [[Category:Article Feedback Blacklist]] as an AFT blacklist category on enwiki' [production]
20:45 <mark> Flipped all backup static routes for LVS service ips from lvs3 to lvs4 [production]
20:42 <mark> lvs3 is now unused. [production]
20:40 <catrope> synchronized php-1.17/resources/startup.js 'touch' [production]
20:40 <catrope> synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 50%' [production]
20:40 <catrope> synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.54%' [production]
20:38 <mark> Migrated apaches LVS service from lvs3 to lvs4 [production]
20:22 <mark> Migrated LVS service 'api' from lvs3 to lvs4 [production]
20:19 <catrope> synchronized php-1.17/resources/startup.js 'touch' [production]
20:19 <catrope> synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 43%' [production]
20:19 <catrope> synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.63%' [production]
20:07 <catrope> synchronized php-1.17/resources/startup.js 'touch' [production]
20:07 <mark> Migrated LVS service 'rendering' from lvs3 to lvs4 [production]
20:05 <catrope> synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 37%' [production]
20:04 <catrope> synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.73%' [production]
19:56 <catrope> synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.css 'r92561' [production]
19:56 <catrope> synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r92561' [production]
19:55 <catrope> synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r92561' [production]
19:55 <catrope> synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r92561' [production]
18:58 <Ryan_Lane> installing nginx with udplogging module on ssl proxies [production]
18:51 <aaron> synchronized php-1.17/maintenance/Maintenance.php 'deploy r92550' [production]
18:27 <Ryan_Lane> pushing change to squids to strip X-Forwarded-Proto, if not set by the ssl proxies [production]
18:08 <Ryan_Lane> repooling sq69 with geoiplookup xff changes, for load testing [production]
18:02 <mark> Fixed LVS on lvs1/lvs2 [production]
17:44 <mark> Disabled RPF check on ae2.1002 on cr1/cr2-eqiad [production]
17:07 <Jeff_Green> rebooting db1025 after dist-upgrade [production]
17:02 <Ryan_Lane> depooling sq69 for geoiplookup tests [production]
16:01 <mark> Enabled BGP on lvs4 and migrated LVS service search_pool3 to lvs4 [production]
15:35 <RobH> rebooting sq63 to check its bios, its arleady down due to other issues and depooled in pybal [production]
15:31 <mark> Added BGP configuration for PyBal to cr1-sdtpa [production]
14:14 <Jeff_Green> commented out puppet-related cron jobs on db1008 as well :-P [production]
13:58 <Jeff_Green> stopping puppet daemon on db1008 to facilitate local my.cnf testing [production]