|
2011-07-20
§
|
| 17:40 |
<RobH> |
lvs3 racked, but powered off. mgmt confirmed working, network port confirmed up |
[production] |
| 16:09 |
<mark> |
Disabling RunCommand monitor on lvs4; not working right |
[production] |
| 15:50 |
<mark> |
Restarted PyBal on lvs4 |
[production] |
| 15:27 |
<RobH> |
powercycling srv283 |
[production] |
| 15:27 |
<RobH> |
srv283 is hung up again, and now wont take my commands in serial console... |
[production] |
| 15:22 |
<RobH> |
srv283 loads are insane, its very slugging, and only serial console works. kicking the apache process over to free up memory and resume normal function |
[production] |
| 14:39 |
<RobH> |
srv239 unresponsive, rebooting |
[production] |
| 14:02 |
<mark> |
Setup nas1 with clustering failover |
[production] |
| 08:44 |
<mark> |
Fixed misc::mediawiki-logger manifest |
[production] |
| 00:04 |
<Ryan_Lane> |
test [[bugzilla:1|bug 1]] |
[production] |
|
2011-07-19
§
|
| 23:59 |
<binasher> |
also set long_query_time to 0.8 on the enwiki read slaves |
[production] |
| 23:09 |
<Ryan_Lane> |
added new ganglia clusters for ssl cluster in pmtpa and esams |
[production] |
| 22:50 |
<binasher> |
reduced long_query_time on db36 to 800ms |
[production] |
| 22:46 |
<binasher> |
setting up mk-query-digest slow query processing on db36 |
[production] |
| 22:43 |
<binasher> |
installing maatkit on core dbs via puppet |
[production] |
| 22:13 |
<Jeff_Green> |
installing mysql 5.0.51a on db1025 to for mydumper/myloader mysqld version upgrade testing |
[production] |
| 21:40 |
<Ryan_Lane> |
make that changing the CNAME to wikipedia-lb |
[production] |
| 21:40 |
<Ryan_Lane> |
changing DNS entry for test.wikipedia.org to wikipedia-lb.wikimedia.org |
[production] |
| 21:04 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 21:03 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Kill underscores in blacklist category name' |
[production] |
| 20:58 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:58 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Set up [[Category:Article Feedback Blacklist]] as an AFT blacklist category on enwiki' |
[production] |
| 20:45 |
<mark> |
Flipped all backup static routes for LVS service ips from lvs3 to lvs4 |
[production] |
| 20:42 |
<mark> |
lvs3 is now unused. |
[production] |
| 20:40 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:40 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 50%' |
[production] |
| 20:40 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.54%' |
[production] |
| 20:38 |
<mark> |
Migrated apaches LVS service from lvs3 to lvs4 |
[production] |
| 20:22 |
<mark> |
Migrated LVS service 'api' from lvs3 to lvs4 |
[production] |
| 20:19 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:19 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 43%' |
[production] |
| 20:19 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.63%' |
[production] |
| 20:07 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:07 |
<mark> |
Migrated LVS service 'rendering' from lvs3 to lvs4 |
[production] |
| 20:05 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 37%' |
[production] |
| 20:04 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.73%' |
[production] |
| 19:56 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.css 'r92561' |
[production] |
| 19:56 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r92561' |
[production] |
| 19:55 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r92561' |
[production] |
| 19:55 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r92561' |
[production] |
| 18:58 |
<Ryan_Lane> |
installing nginx with udplogging module on ssl proxies |
[production] |
| 18:51 |
<aaron> |
synchronized php-1.17/maintenance/Maintenance.php 'deploy r92550' |
[production] |
| 18:27 |
<Ryan_Lane> |
pushing change to squids to strip X-Forwarded-Proto, if not set by the ssl proxies |
[production] |
| 18:08 |
<Ryan_Lane> |
repooling sq69 with geoiplookup xff changes, for load testing |
[production] |
| 18:02 |
<mark> |
Fixed LVS on lvs1/lvs2 |
[production] |
| 17:44 |
<mark> |
Disabled RPF check on ae2.1002 on cr1/cr2-eqiad |
[production] |
| 17:07 |
<Jeff_Green> |
rebooting db1025 after dist-upgrade |
[production] |
| 17:02 |
<Ryan_Lane> |
depooling sq69 for geoiplookup tests |
[production] |
| 16:01 |
<mark> |
Enabled BGP on lvs4 and migrated LVS service search_pool3 to lvs4 |
[production] |
| 15:35 |
<RobH> |
rebooting sq63 to check its bios, its arleady down due to other issues and depooled in pybal |
[production] |