Home » Mailing list archives » git-commits-head » 2008 » December » 29

powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
[view in full thread]
From: Linux Kernel Mailing List
Subject: powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD
Date: Sunday, December 28, 2008 - 6:00 pm

Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=25d6e2...
Commit:     25d6e2d7c58ddc4a3b614fc5381591c0cfe66556
Parent:     4ec577a28980a0790df3c3dfe9c81f6e2222acfb
Author:     Mark Nelson <markn@au1.ibm.com>
AuthorDate: Mon Oct 27 00:46:51 2008 +0000
Committer:  Paul Mackerras <paulus@samba.org>
CommitDate: Wed Nov 5 22:08:29 2008 +1100

    powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD
    
    Update memcpy() to add two new feature sections: one for aligning the
    destination before copying and one for copying using aligned load
    and store doubles.
    
    These new feature sections will only affect Power6 and Cell because
    the CPU feature bit was only added to these two processors.
    
    Power6 gets its best performance in memcpy() when aligning neither the
    source nor the destination, while Cell gets its best performance when
    just the destination is aligned. But in order to save on CPU feature
    bits we can use the previously added CPU_FTR_CP_USE_DCBTZ feature bit
    to differentiate between Power6 and Cell (because CPU_FTR_CP_USE_DCBTZ
    was added to Cell but not Power6).
    
    The first feature section acts to nop out the branch that takes us to
    the code that aligns us to an eight byte boundary for the destination.
    We only want to nop out this branch on Power6.
    
    So the ALT_FTR_SECTION_END() for this feature section creates a test
    mask of the two feature bits ORed together and provides an expected
    result of just CPU_FTR_UNALIGNED_LD_STD, thus we nop out the branch
    if we're on a CPU that has CPU_FTR_UNALIGNED_LD_STD set and
    CPU_FTR_CP_USE_DCBTZ unset.
    
    For the second feature section added, if we're on a CPU that has the
    CPU_FTR_UNALIGNED_LD_STD bit set then we don't want to do the copy
    with aligned loads and stores (and the appropriate shifting left and
    right instructions), so we want to nop out the branch to
    .Lsrc_unaligned.
    
    The andi. used for this branch is moved to just above the branch
    because this allows us to nop out both instructions with just one
    feature section which gives us better performance and doesn't hurt
    readability which two separate feature sections did.
    
    Moving the andi. to just above the branch doesn't have any noticeable
    negative effect on the remaining 64bit processors (the ones that
    didn't have this feature bit added).
    
    On Cell this simple modification results in an improvement to measured
    memcpy() bandwidth of up to 50% in the hot cache case and up to 15% in
    the cold cache case.
    
    On Power6 we get memory bandwidth results that are up to three times
    faster in the hot cache case and up to 50% faster in the cold cache
    case.
    
    Commit 2a9294369bd020db89bfdf78b84c3615b39a5c84 ("powerpc: Add new CPU
    feature: CPU_FTR_CP_USE_DCBTZ") was where CPU_FTR_CP_USE_DCBTZ was
    added.
    
    To say that Cell gets its best performance in memcpy() with just the
    destination aligned is true but only for the reason that the indirect
    shift and rotate instructions, sld and srd, are microcoded on Cell.
    This means that either the destination or the source can be aligned,
    but not both, and seeing as we get better performance with the
    destination aligned we choose this option.
    
    While we're at it make a one line change from cmpldi r1,... to
    cmpldi cr1,... for consistency.
    
    Signed-off-by: Mark Nelson <markn@au1.ibm.com>
    Signed-off-by: Paul Mackerras <paulus@samba.org>
---
 arch/powerpc/lib/memcpy_64.S |   16 ++++++++++++++--
 1 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S
index 3f13112..fe2d34e 100644
--- a/arch/powerpc/lib/memcpy_64.S
+++ b/arch/powerpc/lib/memcpy_64.S
@@ -18,11 +18,23 @@ _GLOBAL(memcpy)
 	andi.	r6,r6,7
 	dcbt	0,r4
 	blt	cr1,.Lshort_copy
+/* Below we want to nop out the bne if we're on a CPU that has the
+   CPU_FTR_UNALIGNED_LD_STD bit set and the CPU_FTR_CP_USE_DCBTZ bit
+   cleared.
+   At the time of writing the only CPU that has this combination of bits
+   set is Power6. */
+BEGIN_FTR_SECTION
+	nop
+FTR_SECTION_ELSE
 	bne	.Ldst_unaligned
+ALT_FTR_SECTION_END(CPU_FTR_UNALIGNED_LD_STD | CPU_FTR_CP_USE_DCBTZ, \
+                    CPU_FTR_UNALIGNED_LD_STD)
 .Ldst_aligned:
-	andi.	r0,r4,7
 	addi	r3,r3,-16
+BEGIN_FTR_SECTION
+	andi.	r0,r4,7
 	bne	.Lsrc_unaligned
+END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD)
 	srdi	r7,r5,4
 	ld	r9,0(r4)
 	addi	r4,r4,-8
@@ -131,7 +143,7 @@ _GLOBAL(memcpy)
 	PPC_MTOCRF	0x01,r6		# put #bytes to 8B bdry into cr7
 	subf	r5,r6,r5
 	li	r7,0
-	cmpldi	r1,r5,16
+	cmpldi	cr1,r5,16
 	bf	cr7*4+3,1f
 	lbz	r0,0(r4)
 	stb	r0,0(r3)
--
To unsubscribe from this list: send the line "unsubscribe git-commits-head" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
Messages in current thread:
powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD, Linux Kernel Mailing ..., (Sun Dec 28, 6:00 pm)