|
2010-11-05
§
|
| 20:08 |
<RobH> |
tested new memcached config, all servers working |
[production] |
| 20:08 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'removed the older servers below srv150 and replaced with tested good new memcached servers' |
[production] |
| 19:55 |
<catrope> |
synchronized images/wikimedia-button.png 'Let's try that with an actual PNG file rather than HTML' |
[production] |
| 19:50 |
<catrope> |
synchronized images/wikimedia-button.png 'New Powered by Wikimedia button' |
[production] |
| 19:11 |
<catrope> |
synchronized php-1.5/skins/common/images/poweredby_mediawiki_88x31.png 'r76126' |
[production] |
| 19:00 |
<rfaulk> |
Added "httpagentparser" Python package on grosley.wikimedia.org from publicly avaialable distutils distribution - this package assists in parsing user-agent header strings found in the 2010/11 fundraiser squid logs |
[production] |
| 18:58 |
<rfaulk> |
Added "setuptools" Python package on grosley.wikimedia.org with apt-get for Wikimedia 2010/11 fundraiser work - This package enables installation of python packages distributed with Python distutils |
[production] |
| 18:22 |
<RobH> |
srv284 is having some booting issues, seems to be harddisk related, but since drac output is slightly garbled, unable to confirm. new rt# 376 |
[production] |
| 17:54 |
<RobH> |
srv284 unresponsive to console, rebooting and fixing it to bring it back into service |
[production] |
| 17:53 |
<RobH> |
srv266 back online and in service |
[production] |
| 17:52 |
<mark> |
Shutdown browne and srv2 for decommissioning - thereby removing the last traces of Fedora from the cluster. Goodbye! |
[production] |
| 17:43 |
<RobH> |
srv266 unresponsive to remote console, rebooting and updating |
[production] |
| 17:42 |
<RobH> |
srv206 fixed, pushed back into lvs |
[production] |
| 17:25 |
<RobH> |
working on srv206, disregard any errors it throws |
[production] |
| 16:40 |
<RobH> |
issue with the new api servers is fixed and they are now back in service |
[production] |
| 16:04 |
<RobH> |
some new api servers are not working right, depooled until they are fixed |
[production] |
| 15:58 |
<mark> |
Removed ibis IPs from Squid ACLs; invalid requests issue has been resolved |
[production] |
| 15:57 |
<mark> |
Fixed NFS mounts on apaches that had them missing since the wikimedia-task-appserver upgrade |
[production] |
| 15:26 |
<RobH> |
working on sq57, disregard flapping |
[production] |
| 15:24 |
<RobH> |
new api apackes srv290-srv301 are online, except srv298 which needs drac correction before installation |
[production] |
| 15:22 |
<RobH> |
dropping old entry for tenwiki in apache config and resyncing/restarting apaches to eliminate error message |
[production] |
| 15:18 |
<RobH> |
pushing srv291-srv301 into lvs |
[production] |
| 15:11 |
<RobH> |
doing puppet runs on srv292-srv301 before pushing them into service |
[production] |
| 14:57 |
<mark> |
Hacked out the 'remotemount' lines in /var/lib/dpkg/info/wikimedia-task-appserver.postrm files to prevent apaches from being without NFS mounts during/between puppet runs and package upgrades |
[production] |
| 14:23 |
<mark> |
Deploying new package wikimedia-task-appserver 1.46 across the cluster, which removes configuration files (now handled by Puppet) |
[production] |
| 11:59 |
<catrope> |
synchronized php-1.5/includes/api/ApiLogin.php 'Revert r76078' |
[production] |
| 11:49 |
<catrope> |
synchronized php-1.5/includes/api/ApiLogin.php 'r76078' |
[production] |
| 05:57 |
<apergos> |
failure booting into be3 on ms4, had to back out. so, no progress, we are back to where we were before the reboots. |
[production] |
| 05:40 |
<apergos> |
cleared up luactivate error, shutdown ms4 again, trying to boot into alt boot environment |
[production] |
| 05:16 |
<apergos> |
used shutdown on ms4, be3 showed as "active on reboot" but it booted into be0 (old boot environment) nonetheless. *grumble* |
[production] |
| 05:06 |
<apergos> |
rebooted ms4 into alt boot environment with current patches applied |
[production] |
| 00:18 |
<RobH> |
new api servers are not coping down the data correctly and not reflecting config changes in puppet, so they fail, srv290+ not online yet |
[production] |
|
2010-11-04
§
|
| 23:06 |
<RobH> |
running puppet across the new api servers srv290-srv301 then will push them in service later when i figure out why they are not doing what I want ;P |
[production] |
| 20:13 |
<RobH> |
sq51 hatees me |
[production] |
| 20:11 |
<RobH> |
new api servers srv290-301 are installed and showing in ganglia, having issues getting the first couple to pool into lvs before i push the rest into service |
[production] |
| 20:09 |
<RobH> |
fixed sq51 |
[production] |
| 19:29 |
<RoanKattouw> |
Strike that, have backed out changes |
[production] |
| 19:06 |
<RoanKattouw> |
Until Mark's made sure they're good, that is |
[production] |
| 19:06 |
<RoanKattouw> |
Changing some files in wmf-deployment/includes/media . DO NOT RUN SCAP or otherwise deploy these changes! |
[production] |
| 18:36 |
<RobH> |
added dns entries for payments |
[production] |
| 17:59 |
<RobH> |
doing puppet runs and final setup for srv290-srv301 |
[production] |
| 16:56 |
<rfaulk> |
Added numpy Python package to grosley.wikimedia.org with apt_get ... For use in the 2010/11 fundraiser to facilitate stats gathering by providing scientific computing functionality in Python |
[production] |
| 16:43 |
<rfaulk> |
Added MySQLdb Python package to on grosley.wikimedia.org with apt-get ... This package will be used to access fundraising databases to facilitate the gathering and synthesis of relevant statistics for the 2010/11 Wikimedia findraiser |
[production] |
| 16:23 |
<mark> |
Set storage1 (varnish) as upload backend on sq41-50, instead of ms4 |
[production] |
| 16:14 |
<RobH> |
sq59 is being bitchy and wont clean the cache, possible hdd issue? will investigate later |
[production] |
| 15:42 |
<RobH> |
sq35 back in rotation |
[production] |
| 15:34 |
<mark> |
Added storage1 (varnish->ms4) as an HTTP backend to sq45's squid config |
[production] |
| 15:34 |
<RobH> |
commenting out sq35, trying to make it work again in pybal |
[production] |
| 15:16 |
<RobH> |
poking at sq59 |
[production] |
| 15:06 |
<RobH> |
sq35 back online, pushed into lvs, partially up - may need to wait up to 5 for idleconnect timer |
[production] |