801-850 of 1529 results (6ms)
2009-01-27 §
13:44 <mark> CARP weight redistribution caused large load spike in upload backend request, causing ms1 overload, probably causing issues on apaches via NFS, etc etc... [production]
13:29 <mark> Lowered CARP weight from 10 to 5 for sq1-10.wikimedia.org, from 15 to 10 for sq11-15 [production]
06:03 <bot broken> retrolog [production]
02:13 <brion> updating extensions/AbuseFilter/Views/AbuseFilterViewList.php (mysql 4 compat issue) [production]
02:04 <brion> installed release versions of mwlib on erzurumi and restarted. these should have updated localizations [production]
01:48 <brion> turning AbuseFilter on on test.... having some mysql 4.0 compat issues. poking [production]
01:47 <brion> srv31 seems very sad; slow/borked login? [production]
01:39 <brion> scapping to update AbuseFilter to current [production]
01:27 <brion> prepping testing of AbuseFilter on test.wikipedia [production]
00:46 <brion> enabling Collection also for de.wikisource per frank's req passed on from community [production]
00:36 <brion> adding NS_HELP to $wgCollectionArticleNamespaces [production]
00:12 <brion> Collection extension being enabled on dewiki [production]
2009-01-26 §
22:39 <RobH> UK Chapter wiki setup per https://bugzilla.wikimedia.org/show_bug.cgi?id=16996 [production]
22:19 <RobH> pushed apache changes for uk chapter wiki [production]
22:13 <RobH> updated dns for uk chapter wiki [production]
19:29 <brion> going to update Collection to current trunk in prep for further activation today [production]
17:01 <RobH> added support for the phone server to dns [production]
2009-01-25 §
12:18 <mark> Announcing routes to AS16265 again [production]
10:17 <domas> our deadlocks are described in X4240 manuals. the fix is either disabling MSI or setting 'options forcedeth max_interrupt_work=15' in modprobe.conf. source: http://docs.sun.com/source/820-2396-14/OSIssues.html [production]
09:31 <domas> db17 live, with 2.6.28.1 kernel [production]
2009-01-24 §
14:53 <domas> db16 and db17 deadlocked: http://p.defau.lt/?A_FG4J__2fq_IXyXWWOXyQ http://p.defau.lt/?XpVljy4JCy1aPECnBv7ilw [production]
11:43 <domas> db17 stuck at nc/tar/kswapd: http://p.defau.lt/?AftWGQnCtD1G80ZjIr7cyg [production]
10:36 <domas> took out db4,db5,db8 for cloning [production]
2009-01-23 §
18:04 <brion> putting load back on db3, it's up to date [production]
17:49 <brion> taking some load off db3 until it catches up [production]
17:46 <brion> also killed a WantedTemplatesPage::recache query which had been running for a day. that ain't sustainable. :P [production]
17:44 <brion> domas restarted morebots a few minutes ago :D [production]
17:43 <brion> syncing update to ApiQueryBacklinks.php with the USE INDEX that was added for this problem [production]
17:41 <brion> killing some stray backlinks queries [production]
2009-01-22 §
00:10 <brion> whitelisting .ott (OpenDocument templates) for private-wiki uploads [production]
2009-01-21 §
20:25 <RobH> some tinkering on http redirects, rollback [production]
17:51 <RobH> setup https for wikitech [production]
17:23 <RobH> setup wikitech to stream weekly backups to tridge [production]
10:29 <domas> db28 powered down because of temperature reading over threshold (45C???) [production]
2009-01-20 §
21:45 <RobH> killed some run away processes on db9 that were killing bugzilla [production]
21:44 <brion> stock long queries on bz again. got rob poking em [production]
20:31 <brion> putting $wgEnotifUseJobQ back for now. change postdates some of the spikes i'm seeing, but it'll be easier to not have to consider it [production]
20:19 <mark> Upgraded kernel to 2.6.24-22 on sq22 [production]
19:57 <brion> disabling $wgEnotifUseJobQ since the lag is ungodly [production]
17:58 <JeLuF> db2 overloaded, error messages about unreachable DB server have been supported. Nearly all connections on DB2 are in status "Sleep" [production]
17:21 <JeLuF> srv154 is reachable again, current load average is 25, no obvious CPU consuming processes visible [production]
17:10 <JeLuF> srv154 went down. Replaced its memcached by srv144's memcached [production]
03:02 <brion> syncing InitialiseSettings -- reenabling CentralNotice which we'd taken temporarily out during the upload breakage [production]
01:50 <Tim> exim4 on lily died while I examined reports of breakage, restarted it [production]
2009-01-19 §
21:28 <mark> Distribution upgrade on lily complete [production]
21:28 <mark> Letting mail through again on lily [production]
21:01 <JeLuF> Bugzilla didn't work. Some long-running (>3h) requests were locking some tables. Killed all long running jobs. [production]
20:05 <mark> Put mail delivery on hold on lily [production]
20:03 <mark> Upgrading lily (Mailing list server) to Ubuntu 8.04 Hardy [production]
14:04 <mark> Set a static ARP entry for 85.17.163.246 on csw1-esams to see if it helps with the inbound packet loss effects [production]