Remove the fields in kmem_cache_cpu that were used to cache data from
kmem_cache when they were in different cachelines. The cacheline that holds
the per cpu array pointer now also holds these values. We can cut down the
struct kmem_cache_cpu size to almost half.
The get_freepointer() and set_freepointer() functions that used to be only
intended for the slow path now are also useful for the hot path since access
to the field does not require accessing an additional cacheline anymore. This
results in consistent use of setting the freepointer for objects throughout
Also we initialize all possible kmem_cache_cpu structures when a slab is
created. No need to initialize them when a processor or node comes online.
And all fields are set to zero. So just use __GFP_ZERO on cpu alloc.
Signed-off-by: Christoph Lameter <firstname.lastname@example.org>
--- linux-2.6.orig/include/linux/slub_def.h 2008-09-19 14:57:50.000000000 -0500
+++ linux-2.6/include/linux/slub_def.h 2008-09-19 14:57:51.000000000 -0500
@@ -36,8 +36,6 @@
void **freelist; /* Pointer to first free per cpu object */
struct page *page; /* The slab from which we are allocating */
int node; /* The node of the page (or -1 for debug) */
- unsigned int offset; /* Freepointer offset (in word units) */
- unsigned int objsize; /* Size of an object (from kmem_cache) */
--- linux-2.6.orig/mm/slub.c 2008-09-19 14:57:50.000000000 -0500
+++ linux-2.6/mm/slub.c 2008-09-19 15:28:00.000000000 -0500
@@ -244,13 +244,6 @@
- * Slow version of get and set free pointer.
- * This version requires touching the cache lines of kmem_cache which
- * we avoid to do in the fast alloc free paths. There we obtain the offset
- * from the page ...