3601-3650 of 5823 results (10ms)
2009-05-26 §
16:32 <Rob> search11 ram upgraded and rebooted. [production]
16:04 <Rob> srv217 shutdown until I can poke it [production]
16:01 <tstarling> synchronized php-1.5/CommonSettings.php 'removed "xx" from $wgStyleVersion' [production]
15:36 <Rob> decommissioning out of warranty and broken servers: srv53, srv67, srv85, srv88, srv90. [production]
15:22 <Rob> db28 fan controller board replaced (not db26, typo) system is now online [production]
15:21 <Rob> db26 fan controller board replaced, system is now online [production]
15:15 <Rob> db26 memory replaced, shows all 32 GB now. Restarting [production]
15:13 <andrew> synchronized php-1.5/includes/Export.php 'Removing useless free()s breaking partial dumps' [production]
14:20 <Rob> shutdown mysql on db26 as it will have the memory replaced in approx. 25 minutes. [production]
13:58 <tstarling> synchronized php-1.5/skins/common/block.js [production]
13:56 <Tim> merged r50871 [production]
13:55 <tstarling> synchronized php-1.5/includes/Block.php [production]
08:14 <Tim> srv78 had apache running but did not have /mnt/upload5 mounted. Upgraded its wikimedia-task-appserver package and mounted it. [production]
2009-05-25 §
21:35 <mark> Modified scap scripts to work on /home-less apaches [production]
21:17 <midom> synchronized php-1.5/db.php 'replaced db16 with db12 for auxiliary roles' [production]
18:51 <mark> Restarted stuck apache on srv187 [production]
18:49 <mark> db16 got in trouble at 18:35, spiking to 2000 threads. Might be related to the nv_nic_irq bug. It recovered 5 mins afterwards [production]
17:33 <mark> Increased COSS cache dirs from 10 GB to 15 GB on knsq16+ esams squids [production]
07:08 <domas> brought williams up, was down, console not responding. [production]
2009-05-23 §
01:07 <tomaszf> restarted srv159 due to hard down [production]
2009-05-22 §
18:03 <mark> Increased big object store on all upload squids from 15 GB to 20 GB [production]
15:24 <mark> Raised max in-memory object size to 100 kiB on all squids [production]
15:11 <mark> Raised cache dir size of large object store on knsq18 from 15 to 20 GB [production]
14:58 <mark> Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB followed by a backend squid restart [production]
14:40 <mark> Increased cache dir size by 50% on knsq16, and upped max object size in memory from 75 to 100 kB [production]
10:41 <mark> apt-get dist-upgrade on searchidx1 [production]
10:29 <mark> Rebooting searchidx1 [production]
09:40 <rainman-sr> zombie java process 1666 on searchidx1 locked in i/o and taking up lots of ram, cannot kill it, searchidx1 need restart [production]
2009-05-21 §
19:09 <Fred> restarted squid process on sq43 as it was not responding properly. [production]
14:08 <Tim> updated [[Switch master]] docs [production]
13:15 <domas> db12 RAID set to no-battery write-behind mode: <code>arcconf SETCACHE 1 LOGICALDRIVE 0 WB noprompt</code> [production]
13:06 <Tim> master switch apparently worked perfectly, in and out of read only mode in like 15 seconds [production]
12:54 <tstarling> synchronized php-1.5/db.php [production]
12:54 <tstarling> synchronized php-1.5/db.php [production]
12:53 <tstarling> synchronized php-1.5/db.php 'setup' [production]
12:50 <Tim> attempting master switch from db12 to db16 (s1/enwiki) using new switch script [production]
12:44 <domas> db12 RAID controller needs battery replacement: http://p.defau.lt/?XsaDOS3KdZHeCO9VzypUnw [production]
04:46 <Tim> added some more singtel proxies to the XFF list [production]
04:45 <tstarling> synchronized php-1.5/extensions/TrustedXFF/trusted-xff.cdb [production]
04:45 <tstarling> synchronized php-1.5/extensions/TrustedXFF/trusted-hosts.txt [production]
2009-05-20 §
21:17 <tomaszf> restarted wikitech db after InnoDB crash [production]
21:12 <Fred> updated all image scaler boxes (srv[43-47,100]) to wikimedia-task-scaler (1.6) [production]
15:50 <Fred> restarted apache on srv224 to bring the load back down from 10. [production]
02:43 <Tim> patched in r50175 to stop Special:RevisionDelete timing on pages with lots of inbound links [production]
02:40 <tstarling> synchronized php-1.5/includes/specials/SpecialRevisiondelete.php [production]
02:06 <tstarling> synchronized php-1.5/includes/specials/SpecialRevisiondelete.php [production]
2009-05-19 §
23:30 <mark> Created ldap/nis/puppet account for ariel [production]
15:45 <robh> synchronized php-1.5/InitialiseSettings.php 'fixing abusefilter settings for zh_yuewiki' [production]
13:36 <Tim> restarted segfaulting apaches on srv212, srv222, srv184 [production]
06:56 <Tim> set cr_status='new' for 1500 recent revisions in code_rev [production]