I use kernel.org kernels, and noticed a bad deterioration in interactive performance between 2.6.27.18 and 2.6.28.7. The newer kernel is really unusable for me on my desktop when the CPUs are loaded. I've got an Intel DP35DP motherboard with a quad-core 64-bit CPU and 8 GB of RAM. I routinely run 4 setiathome jobs at niceness 19. In the past, the setiathome jobs just ran without my really being aware of them. With the new kernel, I can see immediately when they start up. I would be using the desktop as normal, and then a cron job would notice the niced jobs weren't running and start them up, and my desktop experience degrades badly within a few seconds. Everything feels sluggish, the mouse doesn't respond quickly, keys don't immediately register in applications. Worse, I get bad key bouncing, I press a key and it registers two, three, or more times. I can't reliably type a single command line to bash, and have to back up to delete duplicated characters. Sometimes as many as ten keypresses are registered from a single press of a key, but only in the newer kernel, and only when the CPUs are engaged. Switching to 2.6.27.18, everything behaves as it used to, the mouse is responsive, as is the keyboard, and the key bouncing has stopped.
Possibly relevant details: the keyboard is a USB one. I'm using the snd-hda-intel driver for sound, and it appears as if the problem is worse when the audio output is in use. Prior to these two kernels, I was using a 2.6.26.8 kernel, and since upgrading to these newer kernels I get a lot of messages of the form:
kernel: usb 2-4: reset high speed USB device using ehci_hcd and address 2
These messages have come on average every 31 minutes over the past few days. If I'm reading that correctly as device 2 on USB bus 4, then according to the data in /proc, that's the keyboard device.
When running with 2.6.26.8, I didn't get that USB message even once in two weeks of log data.
It's a kernel bug, but yours is odd
Yours is odd because I started seeing this issue arise in 2.6.18, two years ago, and it's way better in 2.6.28. Although, I am uncertain which exact 2.6.28 version I have, as I am using Gentoo and haven't tried the 2.6.28 vanilla on this particular machine. But, I did try 2.6.28.7 vanilla I believe, on my other notebook which appears to have identical hardware, except the hard drive is different.
BTW, I have Intel chipsets in a lot of stuff as well.
Anyhow, I'm thinking your problem is here...
http://bugzilla.kernel.org/show_bug.cgi?id=12309
I think this may be a tough one to nail down because everyone's symptoms are slightly different. Some switch to the AS IO scheduler, and everything is peachy. For others, like myself, I see no difference. But, if you want, you can try this...
cat /sys/block/sda/queue/scheduler
echo anticipatory > /sys/block/sda/queue/scheduler
That worked for my brother, on his machine. He has the Intel ICH8 family of chipsets.