kswapd high CPU usage with no swap

Previous thread: [PATCH 2/2] powerpc: FULL_REGS on exec by Roland McGrath on Monday, September 24, 2007 - 4:52 pm. (2 messages)

Next thread: Linux 2.6.23-rc8 by Linus Torvalds on Monday, September 24, 2007 - 5:46 pm. (4 messages)
From: Jan Kundrát
Date: Monday, September 24, 2007 - 5:13 pm

Hi folks,
I use a 2.6.22-gentoo-r2 SMP kernel with fglrx 8.40.4 [1], tp_smapi-0.32
and ipw3945-1.2.0 on a Thinkpad T60 with dual core Intel Core CPU. My
root filesystem is XFS stored on an internal SATA disk, and I have 1GB
of RAM and no swap.

After several suspend to RAM/resume cycles, the X interface got pretty
slow today. Looking at the top output, I see that one core was
completely busy in the "wa" state and according to `ps auxww`, the
kswapd0 process was in uninterruptable deep sleep.

I don't use any swap because my applications rarely need more memory
than I have in my system and using 1GB of swap yielded poor performance
when it was actually accessed. I know I can tweak VM's preferences, but
I simply don't feel the need to use swap when I have "plenty" of memory.

That said, I've googled a thread [2] which recommends using at least a
small swap because VM somehow performs better with it.

So I created a small (12MB) swap on a physical partition on the same
SATA disk and swapon-ed it. The swap got full immediately but the load
didn't get lower. After that, I've turned it off again, adjusted its
size to be about 1GB and enabled it again. Several tens of seconds
later, swap usage was at approximately 350MB and RAM usage at about
60MB. There was no unusual activity at the background (no intensive
cronned jobs, no locate, nothing, just an idle KDE session with bunch of
idling applications). Touching the swappiness value (0, 10, 40, 60 and
100) didn't change anything on the fact that switching from one
application to another was slow, and even moving a mouse cursor
(USB-attached mouse) in X was delayed and not smooth.

Could you please provide a hint about what might cause such problems,
what am I doing wrong and how can I avoid it? If there are any details
like kernel config that are relevant here, I'll happily provide them.
I'd also appreciate if you keep me on the CC as I'm not subscribed to
this list. If this is not an appropriate place to ask, please accept ...
From: Rik van Riel
Date: Monday, September 24, 2007 - 7:34 pm

On Tue, 25 Sep 2007 02:13:42 +0200

How much memory did you have in "cached" when you looked
with top (and no swap enabled) ?

If the amount of "cached" memory is very low, it could mean
that your shared libraries are being pushed out of memory,
instead of the kernel swapping out some page that belongs to
only one process.

As for kswapd getting into uninterruptible sleep state, it
will do that all by itself sometimes, without even having
any disk IO going on... that code looks a little suspect,
I will look into it.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
-

From: Jan Kundrát
Date: Tuesday, September 25, 2007 - 3:13 am

Hi Rik,
it was pretty low number (several thousands, or maybe tens of thousands).=


In the meanwhile, I've come across your patch [1] ("prevent kswapd from
freeing excessive amounts of lowmem") and applied it locally. I'll see
if it fixes my problem, but at a first glance, it seems that it might
actually slow other things down -- when I switch windows, konqueror's
screen redrawing seems to be pretty slow and I can see it progressing
from top to bottom...

[1] http://lkml.org/lkml/2007/9/5/289

Cheers,
-jkt

--=20
cd /local/pub && more beer > /dev/mouth

From: Rik van Riel
Date: Tuesday, September 25, 2007 - 6:31 pm

On Tue, 25 Sep 2007 12:13:41 +0200

Could you try out the attached patch, too?

Kswapd and try_to_free_pages() have a built-in pause, where
it waits for IO to complete.  However, the current code also
calls blk_congestion_wait() when there is no IO in flight!

This patch should only make the pageout code wait for IO when
there actually is a significant amount of pageout IO in flight.

Signed-off-by: Rik van Riel <riel@redhat.com>
From: Jan Kundrát
Date: Wednesday, September 26, 2007 - 12:27 am

Sorry, I wasn't able to apply it against 2.6.22-gentoo-r3 and vanilla
2.6.22.7; I don't have the "order" member in the "struct scan_control"
and also the bit about "if (sync_writeback =3D=3D PAGEOUT_IO_SYNC &&
may_enter_fs)" was missing so there was no "else" branch to add (and it
seemd pretty significant to me). What version should I patch against?

When this is sorted out, should I keep the previous patch [1] applied as
well?

[1] http://www.redhat.com/archives/nahant-list/2006-March/msg00033.html

Cheers,
-jkt

--=20
cd /local/pub && more beer > /dev/mouth

From: Rik van Riel
Date: Wednesday, September 26, 2007 - 8:25 am

On Wed, 26 Sep 2007 09:27:36 +0200


That doesn't hurt.

-- 
All Rights Reversed
-

From: Jan Kundrát
Date: Wednesday, September 26, 2007 - 3:37 pm

OK, I've used just the latter patch (because I somehow believe the first
one lowers the probability of bad behavior), so let's see if kswapd
consumes CPU again. I don't have any test patter to trigger that CPU
usage, though...

Cheers and thanks for your time,
-jkt

--=20
cd /local/pub && more beer > /dev/mouth

Previous thread: [PATCH 2/2] powerpc: FULL_REGS on exec by Roland McGrath on Monday, September 24, 2007 - 4:52 pm. (2 messages)

Next thread: Linux 2.6.23-rc8 by Linus Torvalds on Monday, September 24, 2007 - 5:46 pm. (4 messages)