|
2009-05-26
§
|
| 16:37 |
<rainman-sr> |
running initial warmup and restart on search11 |
[production] |
| 16:32 |
<Rob> |
search11 ram upgraded and rebooted. |
[production] |
| 16:04 |
<Rob> |
srv217 shutdown until I can poke it |
[production] |
| 16:01 |
<tstarling> |
synchronized php-1.5/CommonSettings.php 'removed "xx" from $wgStyleVersion' |
[production] |
| 15:36 |
<Rob> |
decommissioning out of warranty and broken servers: srv53, srv67, srv85, srv88, srv90. |
[production] |
| 15:22 |
<Rob> |
db28 fan controller board replaced (not db26, typo) system is now online |
[production] |
| 15:21 |
<Rob> |
db26 fan controller board replaced, system is now online |
[production] |
| 15:15 |
<Rob> |
db26 memory replaced, shows all 32 GB now. Restarting |
[production] |
| 15:13 |
<andrew> |
synchronized php-1.5/includes/Export.php 'Removing useless free()s breaking partial dumps' |
[production] |
| 14:20 |
<Rob> |
shutdown mysql on db26 as it will have the memory replaced in approx. 25 minutes. |
[production] |
| 13:58 |
<tstarling> |
synchronized php-1.5/skins/common/block.js |
[production] |
| 13:56 |
<Tim> |
merged r50871 |
[production] |
| 13:55 |
<tstarling> |
synchronized php-1.5/includes/Block.php |
[production] |
| 08:14 |
<Tim> |
srv78 had apache running but did not have /mnt/upload5 mounted. Upgraded its wikimedia-task-appserver package and mounted it. |
[production] |
|
2009-05-22
§
|
| 18:03 |
<mark> |
Increased big object store on all upload squids from 15 GB to 20 GB |
[production] |
| 15:24 |
<mark> |
Raised max in-memory object size to 100 kiB on all squids |
[production] |
| 15:11 |
<mark> |
Raised cache dir size of large object store on knsq18 from 15 to 20 GB |
[production] |
| 14:58 |
<mark> |
Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB followed by a backend squid restart |
[production] |
| 14:40 |
<mark> |
Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB |
[production] |
| 10:41 |
<mark> |
apt-get dist-upgrade on searchidx1 |
[production] |
| 10:29 |
<mark> |
Rebooting searchidx1 |
[production] |
| 09:40 |
<rainman-sr> |
zombie java process 1666 on searchidx1 locked in i/o and taking up lots of ram, cannot kill it, searchidx1 need restart |
[production] |
|
2009-05-21
§
|
| 19:09 |
<Fred> |
restarted squid process on sq43 as it was not responding properly. |
[production] |
| 14:08 |
<Tim> |
updated [[Switch master]] docs |
[production] |
| 13:15 |
<domas> |
db12 RAID set to no-battery write-behind mode: <code>arcconf SETCACHE 1 LOGICALDRIVE 0 WB noprompt</code> |
[production] |
| 13:06 |
<Tim> |
master switch apparently worked perfectly, in and out of read only mode in like 15 seconds |
[production] |
| 12:54 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 12:54 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 12:53 |
<tstarling> |
synchronized php-1.5/db.php 'setup' |
[production] |
| 12:50 |
<Tim> |
attempting master switch from db12 to db16 (s1/enwiki) using new switch script |
[production] |
| 12:44 |
<domas> |
db12 RAID controller needs battery replacement: http://p.defau.lt/?XsaDOS3KdZHeCO9VzypUnw |
[production] |
| 04:46 |
<Tim> |
added some more singtel proxies to the XFF list |
[production] |
| 04:45 |
<tstarling> |
synchronized php-1.5/extensions/TrustedXFF/trusted-xff.cdb |
[production] |
| 04:45 |
<tstarling> |
synchronized php-1.5/extensions/TrustedXFF/trusted-hosts.txt |
[production] |