|
2011-07-21
§
|
| 16:18 |
<mark> |
Deployed lvs3 as backup LVS balancer for 'low-traffic' class LVS services, backing up lvs4 |
[production] |
| 15:22 |
<RobH> |
lvs3 reinstalled, puppet not run |
[production] |
| 15:07 |
<RobH> |
updated dns again for more changes pertaining to lvs |
[production] |
| 15:02 |
<RobH> |
updated dns to add lvs3.wikimedia.org to forward dns, was already in reverse file. |
[production] |
| 14:56 |
<RobH> |
lvs6 reinstalled, puppet not synced and not signed as it may need further work |
[production] |
| 14:50 |
<mark> |
Added new service IPs for bits and upload to varnish and squid classes in puppet |
[production] |
| 14:43 |
<mark> |
Allocated out of subnet IPs for bits and upload.pmtpa |
[production] |
| 14:35 |
<RobH> |
lvs3, lvs5, & lvs6 being (re)installed and setup, please disregard any warnings. |
[production] |
| 14:34 |
<mark> |
Redeployed RunCommand monitor on lvs4, now fixed |
[production] |
| 14:33 |
<RobH> |
lvs3 being reinstalled |
[production] |
|
2011-07-20
§
|
| 22:51 |
<Ryan_Lane> |
started memcached on srv260 |
[production] |
| 21:53 |
<Ryan_Lane> |
added monitoring for esams and pmtpa -lb addresses |
[production] |
| 21:10 |
<Jeff_Green> |
db10 mysqldump is done, replication restarted |
[production] |
| 21:05 |
<Ryan_Lane> |
adding lvs IPs for all esams -lb addresses to amslvs servers |
[production] |
| 20:54 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:52 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 70%' |
[production] |
| 20:52 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.386%' |
[production] |
| 20:47 |
<Ryan_Lane> |
added all new -lb services to esams pybal |
[production] |
| 20:45 |
<Ryan_Lane> |
made esams pybal files consistent. restarting pybal on each amslvs system, in order |
[production] |
| 20:39 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:37 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 63%' |
[production] |
| 20:37 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.428571%' |
[production] |
| 20:25 |
<catrope> |
synchronized php-1.17/resources/startup.js 'touch' |
[production] |
| 20:24 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 57%' |
[production] |
| 20:22 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking rate to 0.47%' |
[production] |
| 19:02 |
<awjrichards> |
synchronized php/extensions/ContactPageFundraiser/SpecialContact.php '[[rev:92653|r92653]]' |
[production] |
| 19:02 |
<awjrichards> |
synchronized php/extensions/ContactPageFundraiser/README '[[rev:92653|r92653]]' |
[production] |
| 19:02 |
<awjrichards> |
synchronized php/extensions/ContactPageFundraiser/install.settings '[[rev:92653|r92653]]' |
[production] |
| 19:01 |
<awjrichards> |
synchronized php/extensions/ContactPageFundraiser/ContactPage.i18n.php '[[rev:92653|r92653]]' |
[production] |
| 19:01 |
<awjrichards> |
synchronized php/extensions/ContactPageFundraiser/ContactPage.alias.php '[[rev:92653|r92653]]' |
[production] |
| 18:59 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php 'Variable-izing enabling of ContactPageFundraiser extension, enabling on testwiki and foundation wiki' |
[production] |
| 18:56 |
<awjrichards> |
synchronized wmf-config/InitialiseSettings.php 'Variable-izing enabling of ContactPageFundraiser extension' |
[production] |
| 18:54 |
<aaron> |
synchronized wmf-config/PrivateSettings.php |
[production] |
| 18:42 |
<AaronSchulz> |
Changed sync-file to be relative to common/ instead of php-1.17/ |
[production] |
| 18:41 |
<aaron> |
synchronizing Wikimedia installation... Revision: 92561: |
[production] |
| 18:41 |
<RobH> |
rebooting the offline sq63 to poll its SAS controller info |
[production] |
| 18:25 |
<Ryan_Lane> |
restarting pybal on lvs1 |
[production] |
| 18:12 |
<Jeff_Green> |
stopping replication on db10 and redoing dump with mysqldump instead of mydumper... |
[production] |
| 18:03 |
<awjrichards> |
synchronized php-1.17/extensions/LandingCheck/SpecialLandingCheck.php '[[rev:92647|r92647]]' |
[production] |
| 18:03 |
<awjrichards> |
synchronized php-1.17/extensions/LandingCheck/LandingCheck.i18n.php '[[rev:92647|r92647]]' |
[production] |
| 17:40 |
<RobH> |
lvs3 racked, but powered off. mgmt confirmed working, network port confirmed up |
[production] |
| 16:09 |
<mark> |
Disabling RunCommand monitor on lvs4; not working right |
[production] |
| 15:50 |
<mark> |
Restarted PyBal on lvs4 |
[production] |
| 15:27 |
<RobH> |
powercycling srv283 |
[production] |
| 15:27 |
<RobH> |
srv283 is hung up again, and now wont take my commands in serial console... |
[production] |
| 15:22 |
<RobH> |
srv283 loads are insane, its very slugging, and only serial console works. kicking the apache process over to free up memory and resume normal function |
[production] |
| 14:39 |
<RobH> |
srv239 unresponsive, rebooting |
[production] |
| 14:02 |
<mark> |
Setup nas1 with clustering failover |
[production] |
| 08:44 |
<mark> |
Fixed misc::mediawiki-logger manifest |
[production] |
| 00:04 |
<Ryan_Lane> |
test [[bugzilla:1|bug 1]] |
[production] |