|
2009-05-26
§
|
| 22:51 |
<river> |
deleted some old snapshots on ms1 |
[production] |
| 22:45 |
<Fred> |
fixed dsh a tad, taking care of a few host which keys have been changed and hosts missing keys... |
[production] |
| 22:28 |
<Fred> |
removed stale amane NFS mount on all apache boxes. |
[production] |
| 22:17 |
<Andrew> |
added 'Change Tagging' component to bugzilla |
[production] |
| 22:05 |
<brion> |
adminned werdna on bugzilla to help w/ component config etc |
[production] |
| 18:17 |
<rainman-sr> |
all of search_4 wikis temporarely moved to search12 server, preparing search11 for taking its place with full set of lucene-search 2.1 features |
[production] |
| 17:52 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18589 Enable AbuseFilter on Cantonese Wikipedia' |
[production] |
| 17:51 |
<robh> |
synchronized php-1.5/abusefilter.php 'updates on bug 18589 Enable AbuseFilter on Cantonese Wikipedia' |
[production] |
| 17:47 |
<robh> |
synchronized php-1.5/InitialiseSettings.php '18938 Please activate Rollback on Simple.Wikibooks' |
[production] |
| 17:27 |
<andrew> |
synchronized php-1.5/lucene.php 'Rainman made me do it! (taking search11 out of rotation)' |
[production] |
| 16:45 |
<rainman-sr> |
running initial warmup and start of lsearchd on search12 |
[production] |
| 16:38 |
<Rob> |
search12 memory upgraded and rebooted |
[production] |
| 16:37 |
<rainman-sr> |
running initial warmup and restart on search11 |
[production] |
| 16:32 |
<Rob> |
search11 ram upgraded and rebooted. |
[production] |
| 16:04 |
<Rob> |
srv217 shutdown until I can poke it |
[production] |
| 16:01 |
<tstarling> |
synchronized php-1.5/CommonSettings.php 'removed "xx" from $wgStyleVersion' |
[production] |
| 15:36 |
<Rob> |
decommissioning out of warranty and broken servers: srv53, srv67, srv85, srv88, srv90. |
[production] |
| 15:22 |
<Rob> |
db28 fan controller board replaced (not db26, typo) system is now online |
[production] |
| 15:21 |
<Rob> |
db26 fan controller board replaced, system is now online |
[production] |
| 15:15 |
<Rob> |
db26 memory replaced, shows all 32 GB now. Restarting |
[production] |
| 15:13 |
<andrew> |
synchronized php-1.5/includes/Export.php 'Removing useless free()s breaking partial dumps' |
[production] |
| 14:20 |
<Rob> |
shutdown mysql on db26 as it will have the memory replaced in approx. 25 minutes. |
[production] |
| 13:58 |
<tstarling> |
synchronized php-1.5/skins/common/block.js |
[production] |
| 13:56 |
<Tim> |
merged r50871 |
[production] |
| 13:55 |
<tstarling> |
synchronized php-1.5/includes/Block.php |
[production] |
| 08:14 |
<Tim> |
srv78 had apache running but did not have /mnt/upload5 mounted. Upgraded its wikimedia-task-appserver package and mounted it. |
[production] |
|
2009-05-22
§
|
| 18:03 |
<mark> |
Increased big object store on all upload squids from 15 GB to 20 GB |
[production] |
| 15:24 |
<mark> |
Raised max in-memory object size to 100 kiB on all squids |
[production] |
| 15:11 |
<mark> |
Raised cache dir size of large object store on knsq18 from 15 to 20 GB |
[production] |
| 14:58 |
<mark> |
Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB followed by a backend squid restart |
[production] |
| 14:40 |
<mark> |
Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB |
[production] |
| 10:41 |
<mark> |
apt-get dist-upgrade on searchidx1 |
[production] |
| 10:29 |
<mark> |
Rebooting searchidx1 |
[production] |
| 09:40 |
<rainman-sr> |
zombie java process 1666 on searchidx1 locked in i/o and taking up lots of ram, cannot kill it, searchidx1 need restart |
[production] |
|
2009-05-21
§
|
| 19:09 |
<Fred> |
restarted squid process on sq43 as it was not responding properly. |
[production] |
| 14:08 |
<Tim> |
updated [[Switch master]] docs |
[production] |
| 13:15 |
<domas> |
db12 RAID set to no-battery write-behind mode: <code>arcconf SETCACHE 1 LOGICALDRIVE 0 WB noprompt</code> |
[production] |
| 13:06 |
<Tim> |
master switch apparently worked perfectly, in and out of read only mode in like 15 seconds |
[production] |
| 12:54 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 12:54 |
<tstarling> |
synchronized php-1.5/db.php |
[production] |
| 12:53 |
<tstarling> |
synchronized php-1.5/db.php 'setup' |
[production] |
| 12:50 |
<Tim> |
attempting master switch from db12 to db16 (s1/enwiki) using new switch script |
[production] |