551-600 of 2614 results (6ms)
2009-05-07 §
21:21 <domas> db30.mgmt needs reset (facilitated by physical movements of power cord) [production]
21:03 <domas> db30 has _second_ disk death [production]
21:00 <domas> db28 FUBAR information: http://p.defau.lt/?EZH6Bg4GwYJDJ4hG3OIJYQ [production]
20:46 <domas> db28 fb0.fm1.f1.speed is flapping between 0 and 21100. needs datacenter inspection and/or vendor service. [production]
19:50 <domas> db19 has corrupted ibdata, depooling [production]
19:50 <midom> synchronized php-1.5/db.php [production]
19:47 <domas> bad disk on db19 actually made I/Os time out, thus corrupting relay logs, reset slave seems to have helped. [production]
19:36 <midom> synchronized php-1.5/db.php 'db25 needs some load' [production]
18:46 <domas> db19 drive failed, needs replacement (you hear, Rob?! :) [production]
17:35 <domas> added retry=1 to ProxyPass for secure.wikimedia apaches backend [production]
16:56 <domas> enabling mod_deflate (bottom of main.conf) on apaches [production]
16:50 <domas> added new singtel subnet to trusted xff [production]
16:50 <midom> synchronized php-1.5/extensions/TrustedXFF/trusted-xff.cdb [production]
07:45 <domas> reset slave on db18 [production]
00:00 <brion> added an 'editor' group to wikitech so we don't have to make all users sysops to edit until we get round to culling the abuse accounts :) [production]
2009-05-06 §
18:48 <Fred> made a couple of changes to the Bayes processing scripts so that they support people moving the Bayes folders around. Wikitech updated. [production]
2009-05-05 §
21:41 <mark> Started Apache on srv32/srv33 [production]
19:11 <mark> Stopping Squid processes on yaseo servers [production]
13:58 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18618 enabling collections on a couple of wikis' [production]
00:34 <tomaszf> upping srv255 to 8 active running dumps [production]
2009-05-04 §
18:43 <tomaszf> spawning 5 extra dumps from srv225 to see throughput of system [production]
18:43 <tomaszf> depooling srv225 from apache node list and adding it as a dumps workers box [production]
10:01 <Tim> set max_connections on ms2 to 2000 using SET GLOBAL, to match the value in /etc/mysql/my.cnf [production]
2009-05-03 §
08:35 <fahadsadah> testing [production]
08:26 <domas> synced http://p.defau.lt/?DO8XAt9PC_rHnVE9nYvRBg to unbreak Special:Log [production]
08:24 <midom> synchronized php-1.5/includes/specials/SpecialLog.php [production]
2009-05-02 §
19:16 <rainman___> search3 mysteriously rebooted around midnight, starting lsearchd on it now [production]
05:46 <Andrew> reports that Special:Log is broken because it's hitting MAX_JOIN_SIZE [production]
00:16 <tomaszf> forcing 644 on dumps using 7za on srv31 until ubuntu bug # 370618 is resolved [production]
2009-05-01 §
19:43 <mark> Unshut peering with AS 2529 on br1-knams [production]
19:11 <tomaszf> added php normalize library to srv31. running a couple batch dumps to test functionality. [production]
17:27 <robh> synchronized php-1.5/InitialiseSettings.php 'updated for usability wiki' [production]
16:49 <Rob> upgraded limesurvey to newest version [production]
16:04 <andrew> synchronized php-1.5/InitialiseSettings.php 'Allow bureaucrats to set/unset inactive right on private wikis -- private request from cary' [production]
13:14 <mark> Installed server sockpuppet [production]
12:59 <andrew> synchronized php-1.5/includes/ChangeTags.php 'Live-merged r50104 -- escaping for classes applied to change tags' [production]
05:47 <tstarling> synchronized php-1.5/includes/UserMailer.php 'merged r49682' [production]
05:36 <andrew> synchronized php-1.5/InitialiseSettings.php 'Re-activated tag filtering with the live patch to ChangeTags.php' [production]
05:32 <andrew> synchronized php-1.5/includes/ChangeTags.php 'Index has been renamed since it was created on Wikimedia' [production]
05:13 <tstarling> synchronized php-1.5/InitialiseSettings.php [production]
05:10 <Tim> $wgUseTagFilter=true experimentally [production]
05:10 <tstarling> synchronized php-1.5/InitialiseSettings.php [production]
04:54 <tstarling> synchronized php-1.5/includes/ChangeTags.php [production]
04:53 <tstarling> synchronized php-1.5/includes/ChangeTags.php [production]
04:50 <Tim> merging r49068 and r49086 [production]
2009-04-30 §
17:10 <Rob> srv143 back online [production]
17:07 <Rob> all memcached back online [production]
17:07 <robh> synchronized php-1.5/mc-pmtpa.php 'swapped out srv142' [production]
17:06 <Rob> srv143 locked up, restarting [production]
17:05 <Rob> srv142 reinstalling [production]