So, I learned a few things since I put up the previous set of benchmarks:
- The erratic behaviour from Linux is due to the glibc memory allocator.

  Using Google's tcmalloc, the problem disappears.
- I missed a few things when porting jemalloc from FreeBSD. One of them

  was fairly major. Due to my mistake jemalloc on NetBSD was, basically,

  single threaded. That said it did show a noticable improvement over

  phkmalloc.
- There was a nasty performance bug in NetBSD's pthread mutexes, which

  is now fixed. libpthread has also had a couple more tweaks for performance

  that have had a positive impact.
- The memory allocator used has a significant effect on sysbench itself:

  it needs to be multithreaded.
- Mindaugas has made more improvements to his scheduler and these are

  showing a really positive effect.
So after making some changes to NetBSD, and changes to how I'm benchmarking

the systems, I have rerun them. In contrast to the previous runs, this one

is done locally:
	http://www.netbsd.org/~ad/sysbench2/4cpu.png 
Kris Kennaway has kindly offered to try NetBSD on an 8-way system. I expect

that NetBSD will hit a fairly clear ceiling due to poll, fcntl and socket

I/O causing contention on kernel_lock. It will be interesting to see.
Thanks,

Andrew

Well you have to be careful there, tcmalloc apparently defers frees, and

is not really a general purpose malloc.  The linux performance problems
I am somewhat surprised by this, because on FreeBSD it is really not

spending much time in the kernel (only ~20% system time), so there does

not seem to be much scope for a 10% performance difference.  Also it

took quite a lot of work to optimize locking of various kernel

subsystems that are used by this workload, and until that point there

was significant kernel lock contention which reduced performance by tens

of percent.  I would have expected this to matter on NetBSD - even with

the vmlocking work there is still more to go.
Here is the initial run with CVS HEAD sources (I took out the obvious

things from GENERIC.MP like I386_CPU support, etc, and removed the

default datasize and stack size limits).  Same benchmark config that

Andrew is using, etc.
   http://people.freebsd.org/~kris/scaling/netbsd.png
There are a couple of things to note:
* the drop-off above 8 threads on FreeBSD is due to non-scalability of

mysql itself.  i.e. it comes from pthread mutex contention in userland.

  This is the only relevant lock contention point in the FreeBSD kernel

on this workload.  There are some things we can do in libpthread to

mitigate the performance loss in the over-contended pthread situation,

but we haven't done them yet.
* The tail end of the graph is somewhat noisy, which is the reason for

the jump at 19 threads (I only graphed a single run).  The distribution

at 20 clients looks like:
+------------------------------------------------------------+

|                                        x  x                |

|x      x   x          xxx   x x  xx  x  x  xxx      x     xx|

|                  |_______________A_M_____________|         |

+------------------------------------------------------------+

     N           Min           Max        Median           Avg     Stddev

x  20       2326.01       2758.86       258...
[ message continues ]

OK, I have repeated the benchmarking in two additional cases:
1) NetBSD with 8 CPUs and some kind of experimental kernel that Andrew

gave me (based on the vmlocking branch).  This is using the new scheduler.
2) As above with experimental libc and libpthread also given to me by

Andrew.  I dunno what changes these contain either :)
I was only able to run in the 8 CPU configuration because when I tried

to disable CPUs with cpuctl, processes would hang under load.  This is

probably a scheduler issue.
   http://people.freebsd.org/~kris/scaling/netbsd.png
This shows some improvement but not much, relatively speaking.  In

particular performance at 4 threads is still significantly below FreeBSD

performance, which (given what I measured previously) suggests that

there is still a performance deficit with 4 CPUs on NetBSD.  It would be

nice to be able to test this directly though, maybe Andrew can give me a

kernel that has MAXCPU=4 or whatever the NetBSD version is.
Kris

It's actually GENERIC.MP from current, with SCHED_M2. No vmlocking code

involved - would you be able to update the labels? The libc has jemalloc,
Interesting. :-). Thanks for running this. I'm still optimistic about the 4

CPU case so I'm very interested in seeing what the results would be. I'll

have a look into the offline problem this evening.
Thanks,

Andrew

In this case, it is not a matter of testing - just left this part for more

thought, while this functionality is not used much. Anyway, I have made a

primary patch for this, hope will be useful shortly.
Switching the CPU offline with cpuctl keeps the execution of bound threads

only. At this moment, it should be OK for the benchmarking.
--

Best regards,

Mindaugas

www.NetBSD.org

Except it didn't work, as above ;-)
Kris

OK thanks.
In the meantime I ran sysbench with postgresql 8.2.  Same NetBSD configs

as before (except I built my own kernel with the sched_m2 patches since

I needed to tweak the sysv ipc parameters).
   http://people.freebsd.org/~kris/scaling/netbsd-pgsql.png
postgresql is much more scalable than mysql on this workload and doesn't

have silly scaling bottlenecks inside the application (cf the tail of

the FreeBSD curve for mysql which is where pthread mutex contention

kicked in).
Kris

Here are some more graphs.
This one is on the 4 CPU P3 500 MHz and shows postgresql 8.2.  FreeBSD

is about 15-20% higher throughput.
   http://people.freebsd.org/~kris/scaling/4cpu-pgsql.png
This one shows mysql on the same system
   http://people.freebsd.org/~kris/scaling/4cpu-mysql.png
In that test NetBSD does outperform FreeBSD but only by 3-4% (in

particular I am not seeing the ~10% difference that Andrew observes on

his 4*p3 700MHz).  Given the age of the hardware and the fact that I am

not seeing it on other workloads or on modern hardware it might just be

due to a small scheduling difference on this configuration.
Kris

Is this system running an amd64 kernel, or an i386 kernel?  One thing I

noticed when comparing NetBSD and FreeBSD performance for a very diferent

workload recently was that NetBSD/amd64 is missing even the most recent

optimized i386 versions of some fairly basic stuff like memset, memcpy,

copyin/copyout, much less versions tuned for modern processors (e.g. for

set and copy SSE2 instructions can be used if you know they're present, as

we do know on amd64).  I suspect FreeBSD/amd64 is better about this.
Thor

It is i386.
Kris

I don't want to get too into linux tuning for this, but horde is a

good alternative to tcmalloc if there are concerns about using

tcmalloc in this capacity.  (horde also works on solaris, and could

probably be ported further)

My opinion is that this is a problem for the Linux and glibc developers

to solve :-)
Kris

-----BEGIN PGP SIGNED MESSAGE-----

Hash: SHA1
Did you mean hoard memory allocator?
Regards

- -----------------------------------------

Adam Hamsik

jabber: haad@jabber.org

icq: 249727910
Proud NetBSD user.
We program to have fun.

Even when we program for money, we want to have fun as well.

~ Yukihiro Matsumoto
-----BEGIN PGP SIGNATURE-----

Version: GnuPG v1.4.7 (Darwin)
iD8DBQFHBkuulIxPgX3Go0MRAmdZAJ93woVHpWwZ2MyG0h8WzFiviHaaigCgsbhG

AGJIrcNHqbknyQ9ipF8DnSo=

=zUED

-----END PGP SIGNATURE-----

Yes.  :)