351-400 of 2425 results (4ms)
2009-05-08 §
18:59 <tfinc> synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php [production]
08:54 <Tim> amane's root partition filled up due to the cp running in a root screen, copying from NFS to an unmounted mount point /mnt/big-disk. Moving some stuff to the real mount point /mnt/scratch (in another screen) [production]
08:25 <Tim> deploying r48837 and r48911 to fix bug 18171 (broken oldimage parameter) [production]
06:11 <midom> synchronized php-1.5/db.php 'got to get coffee' [production]
05:56 <midom> synchronized php-1.5/db.php [production]
05:55 <midom> synchronized php-1.5/db.php [production]
00:08 <brion> db12 no longer overloaded with 'too many connections'. very mysterious [production]
00:04 <brion> db12 [production]
00:04 <brion> db errs on en. poking... [production]
2009-05-07 §
21:36 <domas> db30 disks are shown online, array degraded after 'arcconf rescan', not sure what that means [production]
21:33 <domas> db19 disk error counts: http://p.defau.lt/?pBtD7HBx1O6IboeB9VgINg (one disk just failed few times entirely, other gets lots of aborts/medium errors, might be related) [production]
21:21 <domas> db30.mgmt needs reset (facilitated by physical movements of power cord) [production]
21:03 <domas> db30 has _second_ disk death [production]
21:00 <domas> db28 FUBAR information: http://p.defau.lt/?EZH6Bg4GwYJDJ4hG3OIJYQ [production]
20:46 <domas> db28 fb0.fm1.f1.speed is flapping between 0 and 21100. needs datacenter inspection and/or vendor service. [production]
19:50 <domas> db19 has corrupted ibdata, depooling [production]
19:50 <midom> synchronized php-1.5/db.php [production]
19:47 <domas> bad disk on db19 actually made I/Os time out, thus corrupting relay logs, reset slave seems to have helped. [production]
19:36 <midom> synchronized php-1.5/db.php 'db25 needs some load' [production]
18:46 <domas> db19 drive failed, needs replacement (you hear, Rob?! :) [production]
17:35 <domas> added retry=1 to ProxyPass for secure.wikimedia apaches backend [production]
16:56 <domas> enabling mod_deflate (bottom of main.conf) on apaches [production]
16:50 <domas> added new singtel subnet to trusted xff [production]
16:50 <midom> synchronized php-1.5/extensions/TrustedXFF/trusted-xff.cdb [production]
07:45 <domas> reset slave on db18 [production]
00:00 <brion> added an 'editor' group to wikitech so we don't have to make all users sysops to edit until we get round to culling the abuse accounts :) [production]
2009-05-06 §
18:48 <Fred> made a couple of changes to the Bayes processing scripts so that they support people moving the Bayes folders around. Wikitech updated. [production]
2009-05-05 §
21:41 <mark> Started Apache on srv32/srv33 [production]
19:11 <mark> Stopping Squid processes on yaseo servers [production]
13:58 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18618 enabling collections on a couple of wikis' [production]
00:34 <tomaszf> upping srv255 to 8 active running dumps [production]
2009-05-04 §
18:43 <tomaszf> spawning 5 extra dumps from srv225 to see throughput of system [production]
18:43 <tomaszf> depooling srv225 from apache node list and adding it as a dumps workers box [production]
10:01 <Tim> set max_connections on ms2 to 2000 using SET GLOBAL, to match the value in /etc/mysql/my.cnf [production]
2009-05-03 §
08:35 <fahadsadah> testing [production]
08:26 <domas> synced http://p.defau.lt/?DO8XAt9PC_rHnVE9nYvRBg to unbreak Special:Log [production]
08:24 <midom> synchronized php-1.5/includes/specials/SpecialLog.php [production]
2009-05-02 §
19:16 <rainman___> search3 mysteriously rebooted around midnight, starting lsearchd on it now [production]
05:46 <Andrew> reports that Special:Log is broken because it's hitting MAX_JOIN_SIZE [production]
00:16 <tomaszf> forcing 644 on dumps using 7za on srv31 until ubuntu bug # 370618 is resolved [production]
2009-05-01 §
19:43 <mark> Unshut peering with AS 2529 on br1-knams [production]
19:11 <tomaszf> added php normalize library to srv31. running a couple batch dumps to test functionality. [production]
17:27 <robh> synchronized php-1.5/InitialiseSettings.php 'updated for usability wiki' [production]
16:49 <Rob> upgraded limesurvey to newest version [production]
16:04 <andrew> synchronized php-1.5/InitialiseSettings.php 'Allow bureaucrats to set/unset inactive right on private wikis -- private request from cary' [production]
13:14 <mark> Installed server sockpuppet [production]
12:59 <andrew> synchronized php-1.5/includes/ChangeTags.php 'Live-merged r50104 -- escaping for classes applied to change tags' [production]
05:47 <tstarling> synchronized php-1.5/includes/UserMailer.php 'merged r49682' [production]
05:36 <andrew> synchronized php-1.5/InitialiseSettings.php 'Re-activated tag filtering with the live patch to ChangeTags.php' [production]
05:32 <andrew> synchronized php-1.5/includes/ChangeTags.php 'Index has been renamed since it was created on Wikimedia' [production]