|
2009-04-14
§
|
| 18:21 |
<Rob> |
memcached had stopped on srv89, restarted. |
[production] |
| 18:16 |
<Rob> |
all fans are good on srv86, bringing back online. |
[production] |
| 18:13 |
<Rob> |
srv86 has temp warnings, shutting down to check fans and such |
[production] |
| 17:59 |
<Rob> |
reinstalling srv90 from FC to ubuntu |
[production] |
| 17:52 |
<Rob> |
replaced bad fan in srv90 |
[production] |
| 17:40 |
<Rob> |
pulling srv90, overheating warnings. |
[production] |
| 17:38 |
<Rob> |
srv85 overheating due to dead fans. server is old and out of warranty, decommissioned but kept on site for parting out. |
[production] |
| 17:10 |
<Rob> |
bringing back up sq1, no memory on hand for upgrading these (They are ddr pc3200, all the spare memory we have is ddr2 or ddr 2700) |
[production] |
| 17:02 |
<Rob> |
pulling sq1 for memory upgrade. |
[production] |
| 16:55 |
<Rob> |
replaced bad patch for search9, LOM functions properly. |
[production] |
| 16:38 |
<Rob> |
Upgraded memory in search1 and search2 to a total of 16GB each (previously 8). |
[production] |
| 16:14 |
<Rob> |
Had to restart wikitech due to OOM issues, again. Perhaps it is time to up the memory in the machine or tweak settings. |
[production] |
| 05:05 |
<Andrew> |
testwiki problem seems to be a squid problem, can get srv123, srv84 to serve the main page with no problems by sending a request through netcat. Trying to connect to rr just gets no response |
[production] |
| 05:01 |
<Andrew> |
testwiki seems fubar, timing out on all pageviews. |
[production] |
| 03:47 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php |
[production] |
| 03:45 |
<Andrew> |
Installing AbuseFilter on alswiki |
[production] |
|
2009-04-09
§
|
| 20:15 |
<JeLuF> |
started "maintenance/importImages.php" upload of the second batch of Fotothek images to commons |
[production] |
| 16:37 |
<tstarling> |
synchronized php-1.5/extensions/SecurePoll/includes/Auth.php |
[production] |
| 14:43 |
<Tim> |
php_admin_flag engine on in the SecurePoll directory |
[production] |
| 14:37 |
<tstarling> |
synchronized php-1.5/extensions/SecurePoll/includes/VotePage.php |
[production] |
| 13:55 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'enabling subpages on fiwikiversity' |
[production] |
| 12:59 |
<tstarling> |
synchronized php-1.5/CommonSettings.php 're-enabled SecurePoll' |
[production] |
| 12:59 |
<tstarling> |
synchronized php-1.5/extensions/SecurePoll/includes/Entity.php |
[production] |
| 12:33 |
<tstarling> |
synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' |
[production] |
| 12:32 |
<tstarling> |
synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' |
[production] |
| 09:04 |
<mark> |
Restored Amsterdam traffic; problem resolved |
[production] |
| 08:07 |
<mark> |
Moving Amsterdam traffic to pmtpa while a power problem at esams is being investigated |
[production] |
| 05:27 |
<JeLuF> |
many hosts in esams not reachable any more. Switch outage? |
[production] |
|
2009-04-07
§
|
| 18:34 |
<Rob> |
restarted memcached on srv116 |
[production] |
| 18:26 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 18307 Autopatrolled permission on enwikisource.' |
[production] |
| 16:36 |
<Rob> |
rmeoved the outdated fundraising blog out of planet |
[production] |
| 16:27 |
<Rob> |
singer is back to normal. whygive is fubar, but no one cares ;] The rest of the services are online and functional. |
[production] |
| 08:28 |
<domas> |
reset srv217 - where did I hear that again. it was hanging on image NFS and had segfaulting apache too. 7 incidents with it in past two months - needs hardware diagnostics |
[production] |
| 01:48 |
<Rob> |
blog.wikimedia.org is now up. singer is kinda mostly fixed, i will finish it in the morning. all sites on it are up. |
[production] |
| 00:16 |
<Rob> |
pushed blog.wikimedia.org out of squid via dns |
[production] |