751-800 of 1488 results (3ms)
2009-01-27 §
16:01 <RobH> srv129 didnt have apache running, synced and started [production]
15:59 <RobH> sq11 back online, cleaned [production]
15:40 <RobH> srv126 back online. possible bad disk, if it crashes again, the disk needs replacement. (it went read only before, which seems to sometimes happen even when the disks are not bad.) [production]
15:25 <RobH> srv76 wont boot up, reinstalling. [production]
15:12 <RobH> srv130 coming back online, updated fstab, synced, putting it back in rotation. [production]
15:05 <RobH> moved ts-array4 to its dedicated ports, now its kate's problem ;] [production]
14:49 <Tim> restarted recompressTracked.php [production]
14:33 <Tim> henbane's disk has been full for 8 days due to donate-campaign.log, starting cleanup [production]
14:18 <Tim> killed recompressTracked.php [production]
13:44 <mark> CARP weight redistribution caused large load spike in upload backend request, causing ms1 overload, probably causing issues on apaches via NFS, etc etc... [production]
13:29 <mark> Lowered CARP weight from 10 to 5 for sq1-10.wikimedia.org, from 15 to 10 for sq11-15 [production]
06:03 <bot broken> retrolog [production]
02:13 <brion> updating extensions/AbuseFilter/Views/AbuseFilterViewList.php (mysql 4 compat issue) [production]
02:04 <brion> installed release versions of mwlib on erzurumi and restarted. these should have updated localizations [production]
01:48 <brion> turning AbuseFilter on on test.... having some mysql 4.0 compat issues. poking [production]
01:47 <brion> srv31 seems very sad; slow/borked login? [production]
01:39 <brion> scapping to update AbuseFilter to current [production]
01:27 <brion> prepping testing of AbuseFilter on test.wikipedia [production]
00:46 <brion> enabling Collection also for de.wikisource per frank's req passed on from community [production]
00:36 <brion> adding NS_HELP to $wgCollectionArticleNamespaces [production]
00:12 <brion> Collection extension being enabled on dewiki [production]
2009-01-26 §
22:39 <RobH> UK Chapter wiki setup per https://bugzilla.wikimedia.org/show_bug.cgi?id=16996 [production]
22:19 <RobH> pushed apache changes for uk chapter wiki [production]
22:13 <RobH> updated dns for uk chapter wiki [production]
19:29 <brion> going to update Collection to current trunk in prep for further activation today [production]
17:01 <RobH> added support for the phone server to dns [production]
2009-01-25 §
12:18 <mark> Announcing routes to AS16265 again [production]
10:17 <domas> our deadlocks are described in X4240 manuals. the fix is either disabling MSI or setting 'options forcedeth max_interrupt_work=15' in modprobe.conf. source: http://docs.sun.com/source/820-2396-14/OSIssues.html [production]
09:31 <domas> db17 live, with 2.6.28.1 kernel [production]
2009-01-24 §
14:53 <domas> db16 and db17 deadlocked: http://p.defau.lt/?A_FG4J__2fq_IXyXWWOXyQ http://p.defau.lt/?XpVljy4JCy1aPECnBv7ilw [production]
11:43 <domas> db17 stuck at nc/tar/kswapd: http://p.defau.lt/?AftWGQnCtD1G80ZjIr7cyg [production]
10:36 <domas> took out db4,db5,db8 for cloning [production]
2009-01-23 §
18:04 <brion> putting load back on db3, it's up to date [production]
17:49 <brion> taking some load off db3 until it catches up [production]
17:46 <brion> also killed a WantedTemplatesPage::recache query which had been running for a day. that ain't sustainable. :P [production]
17:44 <brion> domas restarted morebots a few minutes ago :D [production]
17:43 <brion> syncing update to ApiQueryBacklinks.php with the USE INDEX that was added for this problem [production]
17:41 <brion> killing some stray backlinks queries [production]
2009-01-22 §
00:10 <brion> whitelisting .ott (OpenDocument templates) for private-wiki uploads [production]
2009-01-21 §
20:25 <RobH> some tinkering on http redirects, rollback [production]
17:51 <RobH> setup https for wikitech [production]
17:23 <RobH> setup wikitech to stream weekly backups to tridge [production]
10:29 <domas> db28 powered down because of temperature reading over threshold (45C???) [production]
2009-01-20 §
21:45 <RobH> killed some run away processes on db9 that were killing bugzilla [production]
21:44 <brion> stock long queries on bz again. got rob poking em [production]
20:31 <brion> putting $wgEnotifUseJobQ back for now. change postdates some of the spikes i'm seeing, but it'll be easier to not have to consider it [production]
20:19 <mark> Upgraded kernel to 2.6.24-22 on sq22 [production]
19:57 <brion> disabling $wgEnotifUseJobQ since the lag is ungodly [production]
17:58 <JeLuF> db2 overloaded, error messages about unreachable DB server have been supported. Nearly all connections on DB2 are in status "Sleep" [production]
17:21 <JeLuF> srv154 is reachable again, current load average is 25, no obvious CPU consuming processes visible [production]