Home » Mailing list archives » linux-kvm » 2010 » August » 27

Re: Swap usage with KVM (and KSM)

Previous thread: Re: [PATCH v5 6/9] KVM: MMU: introduce pte_prefetch_topup_memory_cache() by Avi Kivity on Sunday, July 11, 2010 - 6:05 am. (2 messages)

Next thread: Re: [PATCH v5 2/9] KVM: MMU: fix race between 'walk_addr' and 'fetch' by Avi Kivity on Sunday, July 11, 2010 - 8:40 am. (1 message)
[view original message]
From: Daniel Bareiro
Subject: Swap usage with KVM
Date: Sunday, July 11, 2010 - 8:12 am

Hi all!

I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
0.12.3 compiled with the source code obtained from the official site of
KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All
this is installed on an HP Proliant DL380 G6 with two Xeon E5530
quadcore processors and 16 GiB of RAM which has two VMs with the
following configuration of memory:

Hostname       |      RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL          |   14 GiB

Initially the host was created with a swap partition of 1 GiB, but today
we found that the use of swap quickly began to grow increasingly.
Therefore, as a contingency, we had to hot-add a logical volume of 1 GB
of swap on the VMHost. Is 'normal' this use of memory?

I copy the Nagios Service Log Entries for the VMHost:

Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
99% free (939 MB out of 956 MB)
10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
79% free (754 MB out of 956 MB)
11-07-2010 00:00:00     11-07-2010 07:08:17     0d 7h 8m 17s    SERVICE OK (HARD)       SWAP OK -
51% free (482 MB out of 956 MB)
11-07-2010 07:08:17     11-07-2010 10:41:07     0d 3h 32m 50s   SERVICE WARNING (HARD)  SWAP WARNING
- 29% free (272 MB out of 956 ...
[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Sunday, July 11, 2010 - 12:12 pm

On Sunday, 11 July 2010 12:12:57 -0300,

Has anyone experienced something like this? Avi? I remember late last
year there was a regression in Linux swapping and Rik and Hugh were
working on it. Are you aware of any?

Thanks in advance for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[ message continues ]
[view original message]
From: Rik van Riel
Subject: Re: Swap usage with KVM
Date: Sunday, July 11, 2010 - 2:49 pm

That depends on what is going on in the host.

Did you notice any performance issues in the guest when

That one should have been fixed.

-- 
All rights reversed
--


[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Monday, August 2, 2010 - 11:57 am

Hi, Rik.

On Sunday, 11 July 2010 17:49:43 -0400,

After the logical volume of 1 GB that I added when I found this problem
(being the operating system with 2 GB), I added other 3 GB to have a
little more margin, but today I got again a new alert of Nagios:

        
Swap usage     WARNING    [...]    SWAP WARNING - 30% free   (1490 MB out of 5052 MB)


I don't see performance issue in the VMs.

Marcelo Tosatti recommended me to apply the EPT patch
(6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
production? Are there plans that this patch is applied on some version
of Linux stable?

Also it can be advisable to update to qemu-kvm 0.12.4 considering what
David Weber said:

http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_...


Thanks for your reply.

Regards,
Daniel
-- 
Mi frase del día:
Collaboration, n.:
	A literary partnership based on the false assumption that the
	other fellow can spell.

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
15:28:17 up 23 days, 20:20, 12 users,  load average: 0.16, 0.14, 0.09

[ message continues ]
[view original message]
From: Rik van Riel
Subject: Re: Swap usage with KVM
Date: Monday, August 2, 2010 - 12:31 pm

Yes, that patch is safe and definitely recommended.

-- 
All rights reversed
--


[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Monday, August 2, 2010 - 12:52 pm

Hi, Rik.

On Monday, 02 August 2010 15:31:47 -0400,

Perfect. Thanks for so fast answer.

And there are some estimates of when this patch is in Linux stable?

Regards,
Daniel
-- 
Mi frase del día:
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
16:44:53 up 23 days, 21:37, 11 users,  load average: 0.02, 0.06, 0.08

[ message continues ]
[view original message]
From: Rik van Riel
Subject: Re: Swap usage with KVM
Date: Monday, August 2, 2010 - 1:22 pm

It should be there already in 2.6.33-stable and 2.6.34-stable.

-- 
All rights reversed
--


[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Monday, August 2, 2010 - 2:43 pm

El lunes 02 de agosto del 2010 a las 16:22:14,

Great! Thanks for your reply.

Regards,
Daniel
-- 
Mi frase del día:
i dont even know if it makes sense at all :) This is an experimental patch
for an experimental kernel :))
		-- Ingo Molnar on linux-kernel

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
18:38:51 up 23 days, 23:31, 11 users,  load average: 0.17, 0.16, 0.10

[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM (and KSM)
Date: Friday, August 27, 2010 - 3:04 am

Hi, Rik.

On Monday, 02 August 2010 16:22:14 -0400,

In a test environment I have Linux 2.6.34.5 compiled with the source
code from kernel.org and qemu-kvm 0.12.4 installed from the Backports
repository of Debian GNU/Linux.

In this environment I have running the following virtual machines:

* 2 x OpenBSD (2 x 512 MB)            |
* 2 x Debian GNU/Linux (2 x 256 MB)   |
* 2 x Debian GNU/Linux (2 x 512 MB)   +--= 11008 MB
* 1 x Centos 5.5 (1 x 256 MB)         |
* 8 x Debian GNU/Linux (8 x 1024 MB)  |

I'm just starting to test KSM, so I was using as reference the read of
the tests made by RedHat about run as many as 52 Windows XP VMs with 1
GB of RAM each on a server with just 16 GB of RAM.

In the previous case the ratio would be 52/16 = 3.25. In my case the
VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH
tests do not talk about the amount of swap used in that case, so I
wonder if a distribution of VMs as I have, it is normal to use 290 MB of
swap.


Regards,
Daniel
-- 
Mi frase del día:
BOFH excuse #335:

the AA battery in the wallclock sends magnetic interference


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
05:57:46 up 48 days, 10:50, 11 users,  load average: 0.01, 0.04, 0.06

[ message continues ]
[view original message]
From: Rik van Riel
Subject: Re: Swap usage with KVM (and KSM)
Date: Friday, August 27, 2010 - 6:42 am

Given that you have 10.75GB worth of virtual machine on a
system with 4GB of RAM, I'd say that 290MB of swap is not
out of the ordinary.

-- 
All rights reversed
--


[ message continues ]
[view original message]
From: Freddie Cash
Subject: Re: Swap usage with KVM
Date: Sunday, July 11, 2010 - 2:05 pm

Are you using virtio drivers in the VMs?

There was an issue with KVM-72 and virtio that leaks memory in the
host until all RAM and swap is used (inside the VMs, no swap is used).
 It was supposed to be fixed in KVM-80-something, though.

Perhaps something similar is happening again?  If you switch the disks
to scsi instead of virtio, does the problem go away?

We are running KVM-72 on Debian 5.0 and have run into this issue.
We'll be upgrading our hosts this month to fix this.

-- 
Freddie Cash
fjwcash@gmail.com
--


[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Sunday, July 11, 2010 - 3:08 pm

Hi, Freddie.

On Sunday, 11 July 2010 14:05:38 -0700,

Yes, we are using Virtio drivers for networking and storage in both VMs
with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
Lenny Backports repositories. For VMHost, we are using a stable version
of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
qemu-kvm 0.12.3 compiled with the source code obtained from the official
site of KVM.

This is the syntax I'm using to boot the virtual machines:


 8587 ?        Sl   6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio
-drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net
nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor
telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait

 9769 ?        Rl   11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio
-drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive
file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio
-drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive
file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio
-drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive
file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net
nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor
telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait


To make the switch from Virtio to SCSI I would have to shut down the
hosts, which would not be a good idea whereas are two productive
systems. At least, before doing so I would be sure of what ...
[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Tuesday, July 20, 2010 - 8:04 am

On Sunday, 11 July 2010 19:08:58 -0300,

This is a current measurement I've taken in both the VMs and in VMHost:

* VMHost:


ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16405140      56448          0       3496      18604
-/+ buffers/cache:   16383040      78548
Swap:      5174220    2401552    2772668


* Aps4:

aps4:~# free
             total       used       free     shared    buffers     cached
Mem:       7164300    7120192      44108          0      23108     239076
-/+ buffers/cache:    6858008     306292
Swap:      2931820      14084    2917736


* Leela:

leela:~# free
             total       used       free     shared    buffers     cached
Mem:       7163836    6905224     258612          0     123380    6282816
-/+ buffers/cache:     499028    6664808
Swap:       979924      35640     944284


As you can see, I added more swap in VMHost for more margin, but
currently only 54% is free.



Thanks in advance for your replies.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[ message continues ]
[view original message]
From: David Weber
Subject: Re: Swap usage with KVM
Date: Tuesday, July 20, 2010 - 12:18 pm

Afaik this should be this bug
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_...

try upgrading to 0.12.4 or backport this commit
http://git.kernel.org/?p=virt/kvm/qemu-
kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960

David
--


[ message continues ]
[view original message]
From: Daniel Bareiro
Subject: Re: Swap usage with KVM
Date: Thursday, July 22, 2010 - 3:09 am

Hi, David.

On Tuesday, 20 July 2010 21:18:09 +0200,

Interesting... I'll try upgrading to qemu-kvm 0.12.4.

Marcelo Tosatti also recommended me to apply this patch I'm attaching.
Someone could confirm whether this is already included in Linux stable?

Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[ message continues ]
Previous thread: Re: [PATCH v5 6/9] KVM: MMU: introduce pte_prefetch_topup_memory_cache() by Avi Kivity on Sunday, July 11, 2010 - 6:05 am. (2 messages)

Next thread: Re: [PATCH v5 2/9] KVM: MMU: fix race between 'walk_addr' and 'fetch' by Avi Kivity on Sunday, July 11, 2010 - 8:40 am. (1 message)