linux/include
Christoph Lameter 4c93c355d5 SLUB: Place kmem_cache_cpu structures in a NUMA aware way
The kmem_cache_cpu structures introduced are currently an array placed in the
kmem_cache struct. Meaning the kmem_cache_cpu structures are overwhelmingly
on the wrong node for systems with a higher amount of nodes. These are
performance critical structures since the per node information has
to be touched for every alloc and free in a slab.

In order to place the kmem_cache_cpu structure optimally we put an array
of pointers to kmem_cache_cpu structs in kmem_cache (similar to SLAB).

However, the kmem_cache_cpu structures can now be allocated in a more
intelligent way.

We would like to put per cpu structures for the same cpu but different
slab caches in cachelines together to save space and decrease the cache
footprint. However, the slab allocators itself control only allocations
per node. We set up a simple per cpu array for every processor with
100 per cpu structures which is usually enough to get them all set up right.
If we run out then we fall back to kmalloc_node. This also solves the
bootstrap problem since we do not have to use slab allocator functions
early in boot to get memory for the small per cpu structures.

Pro:
	- NUMA aware placement improves memory performance
	- All global structures in struct kmem_cache become readonly
	- Dense packing of per cpu structures reduces cacheline
	  footprint in SMP and NUMA.
	- Potential avoidance of exclusive cacheline fetches
	  on the free and alloc hotpath since multiple kmem_cache_cpu
	  structures are in one cacheline. This is particularly important
	  for the kmalloc array.

Cons:
	- Additional reference to one read only cacheline (per cpu
	  array of pointers to kmem_cache_cpu) in both slab_alloc()
	  and slab_free().

[akinobu.mita@gmail.com: fix cpu hotplug offline/online path]
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Cc: "Pekka Enberg" <penberg@cs.helsinki.fi>
Cc: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 09:43:01 -07:00
..
acpi
asm-alpha
asm-arm Merge branch 'devel' of master.kernel.org:/home/rmk/linux-2.6-arm 2007-10-15 16:08:50 -07:00
asm-avr32 x86: optimize page faults like all other achitectures and kill notifier cruft 2007-10-16 09:42:50 -07:00
asm-blackfin Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input 2007-10-15 13:41:39 -07:00
asm-cris
asm-frv frv: missing casts in cmpxchg() 2007-10-14 12:41:51 -07:00
asm-generic flush icache before set_pte() on ia64: flush icache at set_pte 2007-10-16 09:42:59 -07:00
asm-h8300
asm-ia64 flush icache before set_pte() on ia64: flush icache at set_pte 2007-10-16 09:42:59 -07:00
asm-m32r
asm-m68k m68k: Export cachectl.h 2007-10-13 09:41:03 -07:00
asm-m68knommu
asm-mips move a few definitions to au1000_xxs1500.c 2007-10-16 09:42:50 -07:00
asm-parisc
asm-powerpc ppc64: SPARSEMEM_VMEMMAP support 2007-10-16 09:42:51 -07:00
asm-ppc
asm-s390 x86: optimize page faults like all other achitectures and kill notifier cruft 2007-10-16 09:42:50 -07:00
asm-sh x86: optimize page faults like all other achitectures and kill notifier cruft 2007-10-16 09:42:50 -07:00
asm-sh64 Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh64-2.6 2007-10-13 09:50:26 -07:00
asm-sparc [SPARC32]: Add irqflags.h to sparc32 and use it from generic code. 2007-10-13 21:53:11 -07:00
asm-sparc64 SPARC64: SPARSEMEM_VMEMMAP support 2007-10-16 09:42:51 -07:00
asm-um
asm-v850
asm-x86 x86_64: SPARSEMEM_VMEMMAP 2M page size support 2007-10-16 09:42:51 -07:00
asm-xtensa
crypto [CRYPTO] sha: Add header file for SHA definitions 2007-10-10 16:55:50 -07:00
keys
linux SLUB: Place kmem_cache_cpu structures in a NUMA aware way 2007-10-16 09:43:01 -07:00
math-emu
media v4l: copy_to_user() is not a good method name 2007-10-13 09:58:59 -07:00
mtd
net [IPV6]: Replace sk_buff ** with sk_buff * in input handlers 2007-10-15 12:50:28 -07:00
pcmcia pcmcia: use DMA_MASK_NONE for the default for all pcmcia devices 2007-10-16 09:42:50 -07:00
rdma
rxrpc
scsi Merge master.kernel.org:/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6 2007-10-15 08:19:33 -07:00
sound
video
xen
Kbuild