Con Kolivas, maintainer of the high performance -ck patchsets [earlier story], contributed some interesting benchmark results that help compare the performance of several Linux kernels under varying loads. To perform the tests he combined compiling the kernel with a handy tool written by Bob Matthews of RedHat, called irman, described by the tool's readme file as an "Interactive Response Measurement and ANalysis thingy".

Con's tests compare four kernels head to head: 2.4.19, 2.4.19-ck7, 2.4.19-ck7-rmap, and 2.4.18-6mdk. The results compare four types of loads, showing in each how long kernel compilation took, and how much CPU remained available to the system.

I've been using a couple of Con's patch sets on my personal desktop computer, trying to get a feel for the differences between the -aa VM and the -rmap VM. What I found from normal usage is that both seem to handle quite nicely until memory gets full, at which point the -rmap VM feels as if it offers much better performance. Looking at Con's recent benchmarking, his test results agree with what I "felt". As it stands, I am quite content with my Linux desktop server currently running 2.4.19-ck7-rmap.

---

From: Con Kolivas
To: linux-kernel mailing list
Subject: System response benchmarks in performance patches
Date: Sat, 14 Sep 2002 02:08:55 +1000

I came up with a very simple way of measuring responsiveness that gives me
numbers that are meaningful to me. What I've done is the old faithful kernel
compile and measured it under different loads to simulate the pc's ability to
perform under various loads. I have so far benchmarked 2.4.19 versus 2.4.19-ck7,
2.4.19-ck7-rmap and 2.4.18-6mdk(mandrake's kernel in 8.2). 2.5.34 has a dead
keyboard for me so I'm unable to test it as yet.

Here is the story so far:

No Load
Kernel			Time		%CPU
2.4.19			1:49.17		98%
2.4.19-ck7		1:47.66		97%
2.4.19-ck7-rmap		1:48.58		98%
2.4.18-6mdk             1:48.18         98%

Memory Load
Kernel			Time		%CPU
2.4.19			2:15.21		78%
2.4.19-ck7		1:55.88		92%
2.4.19-ck7-rmap		2:18.55		79%
2.4.18-6mdk             2:15.68         79%

IO Load
Kernel			Time		%CPU
2.4.19			3:00.76		58%
2.4.19-ck7		2:01.68		86%
2.4.19-ck7-rmap		2:05.95		83%
2.4.18-6mdk             3:01.48         58%

Process Load
Kernel			Time		%CPU
2.4.19			2:09.42		80%
2.4.19-ck7		1:53.52		92%
2.4.19-ck7-rmap		1:54.39		93%
2.4.18-6mdk             2:10.57         80%

Kernel compiles were done on the same config kernel, fresh boot etc.. on a
single PIII 1133 with make -j 4

The loads were taken from BMatthew's iman found here:
http://people.redhat.com/bmatthews/irman/

Unlike the original program I am not looking at average latencies (which by the
way are http://kernel.kolivas.net

I have yet to merge compressed cache fully without bugs, so -ck8 is still not
finished but R. De Castro is working hard to help me do it :)

Please send me your comments and please cc me to ensure I get your email.

Con Kolivas

From: Rik van Riel
Subject: Re: System response benchmarks in performance patches
Date: Fri, 13 Sep 2002 13:22:54 -0300 (BRT)

On Sat, 14 Sep 2002, Con Kolivas wrote:

> I came up with a very simple way of measuring responsiveness that gives
> me numbers that are meaningful to me. What I've done is the old faithful
> kernel compile and measured it under different loads to simulate the
> pc's ability to perform under various loads.

Absolutely wonderful. I'd love to see this easily scriptable
so we can just run it with one command, eg:

$ ./contest
Kernel Time %CPU 2.4.19 3:00.76 58% 2.4.19-ck7 2:01.68 86% 2.4.19-ck7-rmap 2:05.95 83% 2.4.18-6mdk 3:01.48 58%

Very interesting results. People benchmarking just one thing
at a time won't get variances anywhere near this big, while
real system workload is pretty much always multitasking.

I think I've finally found a benchmark that gives results which
are meaningful in the context of a multitasking system.

regards,

Rik
--
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/ http://distro.conectiva.com/

---

From: Andrew Morton
Subject: Re: System response benchmarks in performance patches
Date: Fri, 13 Sep 2002 13:10:28 -0700

Con Kolivas wrote:
>
> I came up with a very simple way of measuring responsiveness that gives me
> numbers that are meaningful to me. What I've done is the old faithful kernel
> compile and measured it under different loads to simulate the pc's ability to
> perform under various loads. I have so far benchmarked 2.4.19 versus 2.4.19-ck7,
> 2.4.19-ck7-rmap and 2.4.18-6mdk(mandrake's kernel in 8.2). 2.5.34 has a dead
> keyboard for me so I'm unable to test it as yet.

Yes, this is a wonderful test. Very real-world, easy to do and it
tickles a few fairly serious performance problems which we have.

> ...
> The loads were taken from BMatthew's iman found here:
> http://people.redhat.com/bmatthews/irman/

I have issues with irman (I think - didn't read the code really
closely).

It appears to always perform file overwrites - seeking over files,
rewriting them.

This tends to cause best-case behaviour in the VM. The affected pages
are tucked up out of the way on the active list and we do quite well.

If instead the background application is writing _new_ files then
everything falls apart.

I'd suggest that you stick with the kernel compile as the workload,
and vary the background activity a bit. Try tiobench.

(oh, and try turning on everything in the `input' menu; that might
get the keyboard working again in 2.5)

---

/*******************************************
*
* Copyright 2001, Red Hat, Inc., all rights reserved.
*
* Author: Bob Matthews [email blocked]
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*
*******************************************/

Q: What is irman?

A: Interactive Response Measurement and ANalysis thingy.

Q: How do I use irman?

A: Cd to /opt/irman and execute irman.

Q: How should I use irman?

A: You should use irman to compare the response times of otherwise
quiescent machines with identical hardware configurations running
different versions of software, or different hardware configurations
running identical versions of software.

Q: Can I use irman to compare two completely different hardware
configurations running two completely different versions of software?

A: Doing so will not generally produce any interesting results except
to tell you that different hardware configurations running different
software configurations tend to perform differently. It is typically
not necessary to perform tests to obtain such knowledge.

Q: How does irman work?

A: Irman works by measuring the IPC time between two processes under
various artificial loads. In the current implementation, we measure
the amount of time required for a process to write a character to a
pipe, have this character read by another process and then echoed back
to the original process over another pipe.

Q: What does the output of irman mean?

A: Irman measures the round-trip time described above several hundred
thousand times. It then computes the minimum, maximum and average of
these times, and outputs this information for each test, along with
the standard deviation of the observed times.

Q: Can I add different types of loads to irman?

A: Yes. Create a stand-alone process which induces the type of load
you are interested in. Ideally, the process should not require any
user interaction, since it is difficult to reproduce exact behavior
for these types of processes. The process should continue to generate
the load you are interested in until it receives a SIGHUP. At that
point, it should clean up and exit.

To make irman use your load-process, add the appropriate definitions
to irman.h and irman.c:fire_up_load().

Q: What types of loads does irman currently run?

A: 1 - Null load.
2 - Memory load - Repeatedly reference 110% of RAM in a pattern
designed to cause cache misses.
3 - IO load - Read and write 1K chunks from random places in a file
using multiple processes.
4 - Process load - Fork and exec N processes, connected in a
unidirectional ring by pipes. Insert M<<N chunks of data into
the ring and pass them around.

Q: How do *you*, the author, use irman?

A: I, the author, use irman to compare the performance of experimental
kernels against known good ones. Typically, I install all the kernels
I am interested in on a single test box. I then sequentially reboot
the machine into each kernel, running irman immediately after the
reboot, and capturing the output of each run in a separate file.
After all the runs are complete, I massage all of the output files
using the Perl script "massage.pl", which is included in the irman
rpm. I then mail the results to my friends and loved ones.

---

Related links:

• irman