Hi,
Just ran some tbench numbers (from dbench-3.04), on a 2 socket, 8

core x86 system, with 1 NUMA node per socket. With kernel 2.6.24-rc2,

comparing slab vs slub allocators.
I run from 1 to 16 client threads, 5 times each, and restarting

the tbench server between every run. I'm just taking the highest

of each of the 5 tests (because the scheduler placement can

sometimes be poor). It's not completely scientific, but from the

graph you can guess it is relatively stable and seems significant.
Summary: slub is consistently slower. When all CPUs are saturated,

it is around 20% slower. Attached is a graph (x is nrclients, y

is throughput MB/s)
If I can help with reproducing it or testing anything, let me know.

I'll be trying out a few other benchmarks too... anything you want

me to test specifically and I can try.
Thanks,

Nick

Damn your misleading subject! I thought this was going to be about

something interesting.
--

Mathematics is the supreme nostalgia of our time.

-

Actually I did test slob as well -- it's competitive with slab and

slub up to about 4 cores, which is nice.

-

You saw the discussion at

http://marc.info/?l=linux-kernel&m=119354245426072&w=2

and the patches / configurations that were posted to address the issues?
Could you try these?

-

On an 8p 2.6.24-rc2 I see even a 50% regression on tbench SLAB vs. SLUB

when specifying 8 threads. 
Interestingly nothing changes the performance numbers regardless of

debugging on or off etc etc. Usually debugging should reduce performance

but nada. May have something to do with the localhost interface? Something

is effectively throttling SLUB here.
2.6.23 SLUB	           2159.62 MB/sec

2.6.24-rc2-slab head SLUB  1260.80 MB/sec
2.6.24 SLUB should be faster than 2.6.23 SLUB. Still trying to figure out

what is going on....
-

commit deea84b0ae3d26b41502ae0a39fe7fe134e703d0 seems to cause a drop

in SLUB tbench performance:
8p x86_64 system:
2.6.24-rc2:

	1260.80 MB/sec
After reverting the patch:

	2350.04 MB/sec
SLAB performance (which is at 2435.58 MB/sec, ~3% better than SLUB) is not

affected by the patch.
Since this is an alignment change it seems that tbench performance is

sensitive to the data layout? SLUB packs data more tightly than SLAB. So

8 byte allocations could result in cacheline contention if adjacent

objects are allocated from different cpus. SLABs minimum size is 32

bytes so the cacheline contention is likely more limited.
Maybe we need to allocate a mininum of one cacheline to the skb head? Or

padd it out to a full cacheline?
commit deea84b0ae3d26b41502ae0a39fe7fe134e703d0

Author: Herbert Xu <herbert@gondor.apana.org.au>

Date:   Sun Oct 21 16:27:46 2007 -0700
    [NET]: Fix SKB_WITH_OVERHEAD calculation
    The calculation in SKB_WITH_OVERHEAD is incorrect in that it can cause

    an overflow across a page boundary which is what it's meant to prevent.

    In particular, the header length (X) should not be lumped together with

    skb_shared_info.  The latter needs to be aligned properly while the header

    has no choice but to sit in front of wherever the payload is.
    Therefore the correct calculation is to take away the aligned size of

    skb_shared_info, and then subtract the header length.  The resulting

    quantity L satisfies the following inequality:
        SKB_DATA_ALIGN(L + X) + sizeof(struct skb_shared_info) <= PAGE_SIZE
    This is the quantity used by alloc_skb to do the actual allocation.
    Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

    Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h

index f93f22b..369f60a 100644

--- a/include/linux/skbuff.h

+++ b/include/linux/skbuff.h

@@ -41,8 +41,7 @@

 #define SKB_DATA_ALIGN(X)      (((X) +...
[ message continues ]

cc'ed linux-netdev
The data should already be cacheline aligned. It is kmalloced, and

with a minimum size of somewhere around 200 bytes on a 64-bit machine.

So it will hit a cacheline aligned kmalloc slab AFAIKS  -- cacheline

interference is probably not the problem. (To verify, I built slub with

minimum kmalloc size set to 32 like slab and it's no real difference)
But I can't see why restricting the allocation to PAGE_SIZE would help

either. Maybe the macros are used in some other areas.
BTW. your size-2048 kmalloc cache is order-1 in the default setup,

wheras kmalloc(1024) or kmalloc(4096) will be order-0 allocations. And

SLAB also uses order-0 for size-2048. It would be nice if SLUB did the

-

You can try to see the effect that order 0 would have by booting with
slub_max_order=0

-

Yeah, that didn't help much, but in general I think it would give

more consistent and reliable behaviour from slub.

-

From: Nick Piggin <nickpiggin@yahoo.com.au>
Just a note that I'm not ignoring this issue, I just don't have time

to get to it yet.
I suspect the issue is about having a huge skb->data linear area for

TCP sends over loopback.  We're likely getting a much smaller

skb->data linear data area after the patch in question, the rest using

the sk_buff scatterlist pages which are a little bit more expensive to

process.
-

No problem. I would like to have helped more, but it's slow going given

my lack of network stack knowledge. If I get any more interesting data,
It didn't seem to be noticeable at 1 client. Unless scatterlist

processing is going to cause cacheline bouncing, I don't see why this

hurts more as you add CPUs?

-

From: Nick Piggin <nickpiggin@yahoo.com.au>
Is your test system using HIGHMEM?
That's one thing the page vector in the sk_buff can do a lot,

kmaps.

-

No, it's an x86-64, so no highmem.
What's also interesting is that SLAB apparently doesn't have this

condition. The first thing that sprung to mind is that SLAB caches

order > 0 allocations, while SLUB does not. However if anything,

that should actually favour the SLUB numbers if network is avoiding

order > 0 allocations.
I'm doing some oprofile runs now to see if I can get any more info.

-

From: Nick Piggin <nickpiggin@yahoo.com.au>
Here are some other things you can play around with:
1) Monitor the values of skb->len and skb->data_len for packets

   going over loopback.
2) Try removing NETIF_F_SG in drivers/net/loopback.c's dev->feastures

   setting.

-

OK, in vanilla kernels, the page allocator definitely shows higher

in the results (than with Herbert's patch reverted).
27516     2.7217  get_page_from_freelist

21677     2.1442  __rmqueue_smallest

20513     2.0290  __free_pages_ok

18725     1.8522  get_pageblock_flags_group             
Just these account for nearly 10% of cycles. __alloc_skb shows up

higher too. free_hot_cold_page() shows a lot lower though, which

might indicate that actually there is more higher order allocation

activity (I'll check that next).
****

SLUB, avg throughput 1548
CPU: AMD64 family10, speed 1900 MHz (estimated)

Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit

mask of 0x00 (No unit mask) count 100000

samples  %        symbol name

94636     9.3609  copy_user_generic_string

38932     3.8509  ipt_do_table

34746     3.4369  tcp_v4_rcv

29539     2.9218  skb_release_data

27516     2.7217  get_page_from_freelist

26046     2.5763  tcp_sendmsg

24482     2.4216  local_bh_enable

22910     2.2661  ip_queue_xmit

22113     2.1873  ktime_get

21677     2.1442  __rmqueue_smallest

20513     2.0290  __free_pages_ok

18725     1.8522  get_pageblock_flags_group

18580     1.8378  tcp_recvmsg

18108     1.7911  __napi_schedule

17593     1.7402  schedule

16998     1.6813  tcp_ack

16102     1.5927  dev_hard_start_xmit

15751     1.5580  system_call

15707     1.5536  net_rx_action

15150     1.4986  __switch_to

14988     1.4825  tcp_transmit_skb

13921     1.3770  kmem_cache_free

13398     1.3253  __mod_timer

13243     1.3099  tcp_rcv_established

13109     1.2967  __tcp_select_window

11022     1.0902  __tcp_push_pending_frames

10732     1.0615  set_normalized_timespec

10561     1.0446  netif_rx

8840      0.8744  netif_receive_skb

7816      0.7731  nf_iterate

7300      0.7221  __update_rq_clock

6683      0.6610  _read_lock_bh

6504      0.6433  sys_recvf...
[ message continues ]

From: Nick Piggin <nickpiggin@yahoo.com.au>
Thanks for all of this data Nick.
So the thing that's being effected here in TCP is

net/ipv4/tcp.c:select_size(), specifically the else branch:
	int tmp = tp->mss_cache;

 ...

		else {

			int pgbreak = SKB_MAX_HEAD(MAX_TCP_HEADER);
			if (tmp >= pgbreak &&

			    tmp <= pgbreak + (MAX_SKB_FRAGS - 1) * PAGE_SIZE)

				tmp = pgbreak;

		}
This is deciding, in 'tmp', how much linear sk_buff space to

allocate.  'tmp' is initially set to the path MSS, which

for loopback is 16K - the space necessary for packet headers.
The SKB_MAX_HEAD() value has changed as a result of Herbert's

bug fix.   I suspect this 'if' test is passing both with and

without the patch.
But pgbreak is now smaller, and thus the skb->data linear

data area size we choose to use is smaller as well.
You can test if this is precisely what is causing the performance

regression by using the old calculation just here in select_size().
Add something like this local to net/ipv4/tcp.c:
#define OLD_SKB_WITH_OVERHEAD(X)	\

	(((X) - sizeof(struct skb_shared_info)) & \

	 ~(SMP_CACHE_BYTES - 1))

#define OLD_SKB_MAX_ORDER(X, ORDER) \

	OLD_SKB_WITH_OVERHEAD((PAGE_SIZE << (ORDER)) - (X))

#define OLD_SKB_MAX_HEAD(X)		(OLD_SKB_MAX_ORDER((X), 0))
And then use OLD_SKB_MAX_HEAD() in select_size().

-

Thanks for the pointer.  Indeed there is a bug in that area.

I'm not sure whether it's causing the problem at hand but it's

certainly suboptimal.
[TCP]: Fix size calculation in sk_stream_alloc_pskb
We round up the header size in sk_stream_alloc_pskb so that

TSO packets get zero tail room.  Unfortunately this rounding

up is not coordinated with the select_size() function used by

TCP to calculate the second parameter of sk_stream_alloc_pskb.
As a result, we may allocate more than a page of data in the

non-TSO case when exactly one page is desired.
In fact, rounding up the head room is detrimental in the non-TSO

case because it makes memory that would otherwise be available to

the payload head room.  TSO doesn't need this either, all it wants

is the guarantee that there is no tail room.
So this patch fixes this by adjusting the skb_reserve call so that

exactly the requested amount (which all callers have calculated in

a precise way) is made available as tail room.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Cheers,

--

Visit Openswan at http://www.openswan.org/

Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>

Home Page: http://gondor.apana.org.au/~herbert/

PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

--

diff --git a/include/net/sock.h b/include/net/sock.h

index 5504fb9..567e468 100644

--- a/include/net/sock.h

+++ b/include/net/sock.h

@@ -1235,14 +1235,16 @@ static inline struct sk_buff *sk_stream_alloc_pskb(struct sock *sk,

 						   gfp_t gfp)

 {

 	struct sk_buff *skb;

-	int hdr_len;
-	hdr_len = SKB_DATA_ALIGN(sk->sk_prot->max_header);

-	skb = alloc_skb_fclone(size + hdr_len, gfp);

+	skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp);

 	if (skb) {

 		skb->truesize += mem;

 		if (sk_stream_wmem_schedule(sk, skb->truesize)) {

-			skb_reserve(skb, hdr_len);

+			/*

+			 * Make sure that we have exactly size bytes

+			 * available to the caller, no more, no less.

+			 */

+			skb_rese...
[ message continues ]

From: Herbert Xu <herbert@gondor.apana.org.au>
Applied and I'll queue it up for -stable too.

-

Well this is likely the result of the SLUB regression. If you allocate an

order 1 page then the zone locks need to be taken. SLAB queues the a

couple of higher order pages and can so serve a couple of requests without

going into the page allocator whereas SLUB has to go directly to the page

allocator for allocate and free. I guess that needs fixing in the page

allocator. Or do I need to add a mechanism to buffer higher order page

allcoations to SLUB?
-

Actually this serves to discourage people from using high-order

allocations which IMHO is a good thing :)
Cheers,

--

Visit Openswan at http://www.openswan.org/

Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>

Home Page: http://gondor.apana.org.au/~herbert/

PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

-

Yeah, it appears this is what happened. The lockless page allocator

fastpath appears on the list and the slowpaths disappear after

Herbert's patches. SLAB is doing its own thing, so it avoids that
Yeah I completely agree. The right fix is in the caller...

The bug / suboptimal allocation would not have been found in tcp

if not for this ;)
-

Good result. Thanks, everyone!

-

From: Nick Piggin <nickpiggin@yahoo.com.au>
This case is a good example to use the next time a stupid thread

starts up about bug reports not being looked into.  To me it's

seems clearly more a matter of the quality of the bug report.

-

This looks like it fixes the problem!
-

From: Nick Piggin <nickpiggin@yahoo.com.au>
Great, thanks for testing.  I'll apply Herbert's patch tomorrow
Yes, I wonder why too.  I bet objects just got packed differently.
There is this fugly "LOOPBACK_OVERHEAD" macro define in

drivers/net/loopback.c that is trying to figure out the

various overheads that we should subtract from the loopback

MTU we use by default.
It's almost guarenteed to be wrong for the way the allocators

work now.

-

The objects are packed tightly in SLUB and SLUB can allocate smaller

objects (minimum is 8 SLAB mininum is 32).
On free a SLUB object goes directly back to the slab where it came from.

We have no queues in SLUB so we use the first word of the object as a

freepointer. In SLAB the objects first go onto queues and then are drained

later into the slab. On free in SLAB there is usually no need to touch the

object itself. The object pointer is simply moved onto the queue (works

well in SMP, in NUMA we have overhead identifying the queue and

overhead due to the number of queues needed).
-

OK, that makes sense. BTW, are you taking advantage of kmalloc's

"quantization" into slabs WRT the linear data area? I wonder if
That brings performance back up!
I wonder why it isn't causing a problem for SLAB...

-

Doesn't help (with vanilla kernel -- Herbert's patch applied).
data_len histogram drops to 0 and goes to len (I guess that's not

surprising).
Performance is pretty similar (ie. not good).
I'll look at allocator patterns next.

-

-