|
2010-11-15
§
|
| 21:01 |
<mark> |
Lowered CARP weight of esams text amssq* squids from 20 to 10, equal to the older knsq* squids |
[production] |
| 20:33 |
<Ryan_Lane> |
setting authdns-scenario normal |
[production] |
| 20:05 |
<RobH> |
current slowdowns reported for folks hitting AMS squids. Moving traffic to US datacenter should fix major slowdowns on !Wikipedia & !Wikimedia |
[production] |
| 20:04 |
<Ryan_Lane> |
setting authdns-scenario esams-down |
[production] |
| 19:56 |
<RobH> |
fixed nagios again |
[production] |
| 19:51 |
<RobH> |
updating dns for new owa processing nodes |
[production] |
| 18:54 |
<RobH> |
srv298 now online in api pool |
[production] |
| 18:21 |
<Ryan_Lane> |
fixing puppet manually on sq34, sq36, sq37, sq39, sq40, and knsq13 |
[production] |
| 18:18 |
<RobH> |
gilman to secure gateway project stalled, needs network checks done |
[production] |
| 18:07 |
<Ryan_Lane> |
puppetizing /etc/default/puppet, since some hosts had START=no, instead of START=yes |
[production] |
| 17:46 |
<RobH> |
gilman needed hard reset, ilom responsive now (thx rich!) |
[production] |
| 17:35 |
<Ryan_Lane> |
restarting puppet again on all nodes using -M flag for ddsh to see system names (checking for errors) |
[production] |
| 17:23 |
<Ryan_Lane> |
restarting puppet on all nodes |
[production] |
| 17:12 |
<RobH> |
sq57 disk replaced, reinstalled, back in service |
[production] |
| 17:09 |
<mark> |
Restarted apache on sockpuppet with concurrency 4 instead of 3 |
[production] |
| 17:04 |
<RobH> |
puppet is now failing to work properly on sq57, why did we upgrade puppet again? |
[production] |
| 16:59 |
<RobH> |
sq57 reinstalled and doing post installation configuration |
[production] |
| 16:40 |
<Ryan_Lane> |
upping configtimeout setting in puppet to 8 minutes, globally |
[production] |
| 16:33 |
<Ryan_Lane> |
trying to add puppet.conf to puppet again |
[production] |
| 16:24 |
<Ryan_Lane> |
undoing puppet.conf changes |
[production] |
| 16:20 |
<RobH> |
sq57 coming down for reinstallation |
[production] |
| 16:19 |
<RobH> |
db13 back online, restarted mysql, but its currently commented out of db.php |
[production] |
| 16:12 |
<RobH> |
not sure why db13 is borked, but its down, poking at it |
[production] |
| 16:09 |
<Ryan_Lane> |
added puppet.conf to puppet. pushing change out |
[production] |
| 16:00 |
<RobH> |
torrus is up again |
[production] |
| 15:59 |
<richcole> |
swaped sq57 sdb bad drive |
[production] |
| 15:56 |
<RobH> |
torrus is down, again, restarting and cleaning up its services |
[production] |
| 15:52 |
<RobH> |
manually purged spence nagios, started manually, working until puppet borks it again |
[production] |
| 15:10 |
<RobH> |
nagios is down, investigating |
[production] |
|
2010-11-14
§
|
| 23:28 |
<mark> |
Fixed Nagios |
[production] |
| 20:00 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25918 - Namespaces on vec.wikisource.org' |
[production] |
| 14:57 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25904 - Create the Swedish Wikiversity (wv/sv)' |
[production] |
| 14:56 |
<jeluf> |
ran sync-common-all '25904 - Create the Swedish Wikiversity (wv/sv)' |
[production] |
| 14:28 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25918 - Namespaces on vec.wikisource.org' |
[production] |
| 08:26 |
<domas> |
ran purge-nagios-resources.py manually to bring up nagios |
[production] |
| 07:14 |
<domas> |
reduced passenger pool size to 4 on sockpuppet |
[production] |
| 04:42 |
<Ryan_Lane> |
moving /etc/nagios/puppet_services.cfg to .bak and rerunning puppet |
[production] |
| 03:05 |
<Ryan_Lane> |
modified nagios puppet manifest to purge decommisioned servers from the services configuration |
[production] |
| 01:48 |
<jeluf> |
synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 01:33 |
<Ryan_Lane> |
temporarily upped configtimeout in /etc/puppet/puppet.conf to 8 minutes on spence so that puppet would run |
[production] |