Re: SLUB performance regression vs SLAB

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Christoph Lameter
Date: Friday, October 5, 2007 - 12:31 pm

On Fri, 5 Oct 2007, Jens Axboe wrote:


In case there is someone who wants to hack on it: Here is what I got so 
far for batching the frees. I will try to come up with a test next week if 
nothing else happens before:

Patch 1/2 on top of mm:

SLUB: Keep counter of remaining objects on the per cpu list

Add a counter to keep track of how many objects are on the per cpu list.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

---
 include/linux/slub_def.h |    1 +
 mm/slub.c                |    8 ++++++--
 2 files changed, 7 insertions(+), 2 deletions(-)

Index: linux-2.6.23-rc8-mm2/include/linux/slub_def.h
===================================================================
--- linux-2.6.23-rc8-mm2.orig/include/linux/slub_def.h	2007-10-04 22:41:58.000000000 -0700
+++ linux-2.6.23-rc8-mm2/include/linux/slub_def.h	2007-10-04 22:42:08.000000000 -0700
@@ -15,6 +15,7 @@ struct kmem_cache_cpu {
 	void **freelist;
 	struct page *page;
 	int node;
+	int remaining;
 	unsigned int offset;
 	unsigned int objsize;
 };
Index: linux-2.6.23-rc8-mm2/mm/slub.c
===================================================================
--- linux-2.6.23-rc8-mm2.orig/mm/slub.c	2007-10-04 22:41:58.000000000 -0700
+++ linux-2.6.23-rc8-mm2/mm/slub.c	2007-10-04 22:42:08.000000000 -0700
@@ -1386,12 +1386,13 @@ static void deactivate_slab(struct kmem_
 	 * because both freelists are empty. So this is unlikely
 	 * to occur.
 	 */
-	while (unlikely(c->freelist)) {
+	while (unlikely(c->remaining)) {
 		void **object;
 
 		/* Retrieve object from cpu_freelist */
 		object = c->freelist;
 		c->freelist = c->freelist[c->offset];
+		c->remaining--;
 
 		/* And put onto the regular freelist */
 		object[c->offset] = page->freelist;
@@ -1491,6 +1492,7 @@ load_freelist:
 
 	object = c->page->freelist;
 	c->freelist = object[c->offset];
+	c->remaining = s->objects - c->page->inuse - 1;
 	c->page->inuse = s->objects;
 	c->page->freelist = NULL;
 	c->node = page_to_nid(c->page);
@@ -1574,13 +1576,14 @@ static void __always_inline *slab_alloc(
 
 	local_irq_save(flags);
 	c = get_cpu_slab(s, smp_processor_id());
-	if (unlikely(!c->freelist || !node_match(c, node)))
+	if (unlikely(!c->remaining || !node_match(c, node)))
 
 		object = __slab_alloc(s, gfpflags, node, addr, c);
 
 	else {
 		object = c->freelist;
 		c->freelist = object[c->offset];
+		c->remaining--;
 	}
 	local_irq_restore(flags);
 
@@ -1686,6 +1689,7 @@ static void __always_inline slab_free(st
 	if (likely(page == c->page && c->node >= 0)) {
 		object[c->offset] = c->freelist;
 		c->freelist = object;
+		c->remaining++;
 	} else
 		__slab_free(s, page, x, addr, c->offset);
 


-
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
[15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Tue Sep 18, 8:36 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Fri Sep 28, 10:33 am)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Fri Sep 28, 11:20 am)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Fri Sep 28, 11:41 am)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 1:50 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 1:55 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 2:01 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 2:10 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 2:38 pm)
Re: [15/17] SLUB: Support virtual fallback via SLAB_VFALLBACK, Christoph Lameter, (Mon Oct 1, 2:52 pm)
SLUB performance regression vs SLAB, Matthew Wilcox, (Thu Oct 4, 9:16 am)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 10:38 am)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 10:49 am)
Re: SLUB performance regression vs SLAB, Arjan van de Ven, (Thu Oct 4, 10:50 am)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 10:58 am)
Re: SLUB performance regression vs SLAB, Peter Zijlstra, (Thu Oct 4, 11:26 am)
Re: SLUB performance regression vs SLAB, Matthew Wilcox, (Thu Oct 4, 11:32 am)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 12:05 pm)
Re: SLUB performance regression vs SLAB, Matthew Wilcox, (Thu Oct 4, 12:28 pm)
Re: SLUB performance regression vs SLAB, Siddha, Suresh B, (Thu Oct 4, 12:46 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Thu Oct 4, 1:48 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Thu Oct 4, 1:55 pm)
Re: SLUB performance regression vs SLAB, Matthew Wilcox, (Thu Oct 4, 1:58 pm)
Re: SLUB performance regression vs SLAB, Chuck Ebbert, (Thu Oct 4, 2:02 pm)
Re: SLUB performance regression vs SLAB, Matthew Wilcox, (Thu Oct 4, 2:05 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Thu Oct 4, 2:05 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Thu Oct 4, 2:11 pm)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 2:11 pm)
Re: SLUB performance regression vs SLAB, Chuck Ebbert, (Thu Oct 4, 2:47 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Thu Oct 4, 3:07 pm)
Re: SLUB performance regression vs SLAB, David Chinner, (Thu Oct 4, 3:23 pm)
RE: SLUB performance regression vs SLAB, David Schwartz, (Thu Oct 4, 4:39 pm)
Re: SLUB performance regression vs SLAB, Chuck Ebbert, (Thu Oct 4, 4:49 pm)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Thu Oct 4, 7:43 pm)
Re: SLUB performance regression vs SLAB, Arjan van de Ven, (Thu Oct 4, 7:53 pm)
RE: SLUB performance regression vs SLAB, David Schwartz, (Thu Oct 4, 9:18 pm)
Re: SLUB performance regression vs SLAB, Jens Axboe, (Thu Oct 4, 11:48 pm)
Re: SLUB performance regression vs SLAB, Pekka Enberg, (Fri Oct 5, 2:19 am)
Re: SLUB performance regression vs SLAB, Jens Axboe, (Fri Oct 5, 2:28 am)
Re: SLUB performance regression vs SLAB, Andi Kleen, (Fri Oct 5, 4:12 am)
Re: SLUB performance regression vs SLAB, Matthew Wilcox, (Fri Oct 5, 4:56 am)
Re: SLUB performance regression vs SLAB, Jens Axboe, (Fri Oct 5, 5:37 am)
Re: SLUB performance regression vs SLAB, Jens Axboe, (Fri Oct 5, 5:39 am)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Fri Oct 5, 12:27 pm)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Fri Oct 5, 12:31 pm)
Re: SLUB performance regression vs SLAB, Christoph Lameter, (Fri Oct 5, 12:32 pm)
Re: SLUB performance regression vs SLAB, Peter Zijlstra, (Fri Oct 5, 1:32 pm)
Re: SLUB performance regression vs SLAB, David Miller, (Fri Oct 5, 2:31 pm)