Re: MMIO and gcc re-ordering issue

!MAILaRCHIVE_VOTE_RePLACE
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Nick Piggin <nickpiggin@...>
Cc: Jes Sorensen <jes@...>, Jeremy Higdon <jeremy@...>, Roland Dreier <rdreier@...>, <benh@...>, Arjan van de Ven <arjan@...>, <linux-arch@...>, <linux-kernel@...>, <tpiepho@...>, <linuxppc-dev@...>, <scottwood@...>, <torvalds@...>, David Miller <davem@...>, <alan@...>
Date: Tuesday, June 3, 2008 - 12:52 pm

On Monday, June 02, 2008 9:33 pm Nick Piggin wrote:

I think you mean wrt cacheable memory accesses here (though iirc on ia64 
spin_unlock has release semantics, so at least it'll barrier other stores).


Well, given how undefined things have been in the past, each arch has had to 
figure out what things mean (based on looking at drivers & core code) then 
come up with appropriate primitives.  On Altix, we went both directions:  we 
made regular PIO reads (readX etc.) *very* expensive to preserve 
compatibility with what existing drivers expect, and added a readX_relaxed to 
give a big performance boost to tuned drivers.

OTOH, given that posted PCI writes were nothing new to Linux, but the Altix 
network topology was, we introduced mmiowb() (with lots of discussion I might 
add), which has clear and relatively simple usage guidelines.

Now, in hindsight, using a PIO write set & test flag approach in 
writeX/spin_unlock (ala powerpc) might have been a better approach, but iirc 
that never came up in the discussion, probably because we were focused on PCI 
posting and not uncached vs. cached ordering.


I agree, but afaik the only change Altix ended up forcing on people was 
mmiowb(), but that turned out to be necessary on mips64 (and maybe some other 
platforms?) anyway.


Aside from the obvious performance impact of making all the readX/writeX 
routines strongly ordered, both in terms of PCI posting and cacheable vs. 
uncacheable accesses, it also makes things inconsistent.  Both core code & 
drivers will still have to worry about regular, cacheable memory barriers for 
correctness, but it looks like you're proposing that they not have to think 
about I/O ordering.

At any rate, I don't think anyone would argue against defining the ordering 
semantics of all of these routines (btw you should also include ordering wrt 
DMA & PCI posting); the question is what's the best balance between keeping 
the driver requirements simple and the performance cost on complex arches.

Jesse
--
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
[PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 4:40 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Andreas Schwab, (Tue May 20, 6:00 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 6:11 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Andreas Schwab, (Tue May 20, 6:47 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 7:14 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Andreas Schwab, (Wed May 21, 4:03 am)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Wed May 21, 4:25 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Tue May 20, 5:16 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 6:00 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Wed May 21, 10:00 am)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Wed May 21, 3:44 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Wed May 21, 4:41 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 6:55 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Wed May 21, 10:01 am)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, David Miller, (Tue May 20, 6:39 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, David Miller, (Tue May 20, 6:53 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Fri May 23, 12:24 am)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Thu May 22, 6:56 pm)
MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Mon May 26, 9:33 pm)
Re: MMIO and gcc re-ordering issue, Jonathan Corbet, (Tue May 27, 11:28 am)
Re: MMIO and gcc re-ordering issue, David Miller, (Mon May 26, 9:40 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Mon May 26, 10:15 pm)
Re: MMIO and gcc re-ordering issue, David Miller, (Mon May 26, 10:28 pm)
Re: MMIO and gcc re-ordering issue, Alan Cox, (Tue May 27, 4:24 am)
Re: MMIO and gcc re-ordering issue, Arjan van de Ven, (Mon May 26, 11:42 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 3:08 am)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 11:50 am)
Re: MMIO and gcc re-ordering issue, Pavel Machek, (Sat May 31, 4:04 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 5:11 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 5:33 pm)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Thu May 29, 10:47 am)
Re: MMIO and gcc re-ordering issue, Jeremy Higdon, (Sat May 31, 3:52 am)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Mon Jun 2, 5:56 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 12:33 am)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Tue Jun 3, 12:52 pm)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Thu Jun 5, 4:40 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Thu Jun 5, 4:43 am)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Thu Jun 12, 11:07 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Thu Jun 12, 8:07 pm)
Re: MMIO and gcc re-ordering issue, Jeremy Higdon, (Tue Jun 3, 4:15 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 4:19 am)
Re: MMIO and gcc re-ordering issue, Jeremy Higdon, (Tue Jun 3, 4:45 am)
Re: MMIO and gcc re-ordering issue, Jeremy Higdon, (Mon Jun 2, 5:02 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Thu May 29, 6:06 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Thu May 29, 6:25 pm)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Thu May 29, 11:56 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Thu May 29, 5:40 pm)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Fri May 30, 5:48 am)
Re: MMIO and gcc re-ordering issue, Pavel Machek, (Sat May 31, 4:14 am)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Mon Jun 2, 5:48 am)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Thu May 29, 5:53 pm)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Fri May 30, 5:39 am)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Thu May 29, 5:48 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Thu May 29, 6:05 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Thu May 29, 9:53 pm)
Re: MMIO and gcc re-ordering issue, James Bottomley, (Thu May 29, 11:01 am)
Re: MMIO and gcc re-ordering issue, Jes Sorensen, (Fri May 30, 5:36 am)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Fri May 30, 1:21 pm)
Re: MMIO and gcc re-ordering issue, Jeremy Higdon, (Sat May 31, 3:57 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 6:13 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 6:39 pm)
Re: MMIO and gcc re-ordering issue, James Bottomley, (Tue May 27, 12:37 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 1:38 pm)
Re: MMIO and gcc re-ordering issue, James Bottomley, (Tue May 27, 1:53 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 2:07 pm)
Re: MMIO and gcc re-ordering issue, Chris Friesen, (Tue May 27, 5:23 pm)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Tue May 27, 7:04 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 5:29 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 2:17 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue May 27, 12:08 am)
Re: MMIO and gcc re-ordering issue, Arjan van de Ven, (Tue May 27, 12:20 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Mon May 26, 11:39 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 11:35 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 5:10 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 5:30 pm)
Re: MMIO and gcc re-ordering issue, Alan Cox, (Tue May 27, 5:38 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 5:59 pm)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Tue May 27, 5:53 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 6:02 pm)
Re: MMIO and gcc re-ordering issue, Alan Cox, (Tue May 27, 5:46 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 5:38 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 5:55 pm)
Re: MMIO and gcc re-ordering issue, Russell King, (Mon Jun 2, 3:24 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 12:16 am)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue Jun 3, 3:43 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue Jun 3, 6:26 pm)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Tue Jun 3, 5:33 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue Jun 3, 5:44 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 10:25 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Wed Jun 4, 2:39 am)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue Jun 3, 10:47 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 10:19 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue Jun 3, 2:47 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue Jun 3, 3:07 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 10, 2:56 am)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Wed Jun 11, 1:20 am)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Wed Jun 11, 12:18 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Wed Jun 11, 1:00 am)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Wed Jun 11, 10:46 am)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Wed Jun 11, 1:13 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Wed Jun 11, 1:35 am)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Thu Jun 12, 8:14 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Thu Jun 12, 9:08 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Wed Jun 11, 2:02 am)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Tue Jun 10, 1:41 pm)
Re: MMIO and gcc re-ordering issue, James Bottomley, (Tue Jun 10, 2:10 pm)
Re: MMIO and gcc re-ordering issue, Roland Dreier, (Tue Jun 10, 3:05 pm)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Tue Jun 10, 3:19 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 10, 11:29 pm)
Re: MMIO and gcc re-ordering issue, Jesse Barnes, (Wed Jun 11, 12:07 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Thu Jun 12, 7:27 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue Jun 10, 11:40 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Wed Jun 11, 12:06 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 10:05 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue Jun 3, 10:46 pm)
Re: MMIO and gcc re-ordering issue, Alan Cox, (Wed Jun 4, 7:47 am)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Tue Jun 3, 2:55 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue Jun 3, 3:57 pm)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Tue Jun 3, 5:35 pm)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue Jun 3, 5:58 pm)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 10:00 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue Jun 3, 12:32 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 2:11 am)
Re: MMIO and gcc re-ordering issue, Paul Mackerras, (Tue Jun 3, 2:53 am)
Re: MMIO and gcc re-ordering issue, Nick Piggin, (Tue Jun 3, 3:18 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue Jun 3, 2:48 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 6:19 pm)
Re: MMIO and gcc re-ordering issue, Arnd Bergmann, (Thu May 29, 3:10 am)
Re: MMIO and gcc re-ordering issue, Alan Cox, (Thu May 29, 6:46 am)
Re: MMIO and gcc re-ordering issue, Matthew Wilcox, (Tue May 27, 5:42 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 6:17 pm)
Re: MMIO and gcc re-ordering issue, Haavard Skinnemoen, (Wed May 28, 4:36 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Thu May 29, 9:13 pm)
Re: MMIO and gcc re-ordering issue, Haavard Skinnemoen, (Fri May 30, 2:07 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Fri May 30, 3:24 am)
Re: MMIO and gcc re-ordering issue, Haavard Skinnemoen, (Fri May 30, 4:27 am)
Re: MMIO and gcc re-ordering issue, Geert Uytterhoeven, (Fri May 30, 5:22 am)
Re: MMIO and gcc re-ordering issue, Haavard Skinnemoen, (Mon Jun 2, 4:11 am)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Wed Jun 4, 11:31 am)
Re: MMIO and gcc re-ordering issue, Scott Wood, (Mon Jun 2, 11:48 am)
Re: MMIO and gcc re-ordering issue, Haavard Skinnemoen, (Tue Jun 3, 3:46 am)
Re: MMIO and gcc re-ordering issue, Pantelis Antoniou, (Thu May 29, 7:05 am)
Re: MMIO and gcc re-ordering issue, Trent Piepho, (Tue May 27, 2:23 pm)
Re: MMIO and gcc re-ordering issue, Scott Wood, (Tue May 27, 2:33 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 12:47 pm)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Tue May 27, 5:12 pm)
Re: MMIO and gcc re-ordering issue, Linus Torvalds, (Tue May 27, 1:31 pm)
Re: MMIO and gcc re-ordering issue, Ingo Molnar, (Mon Jun 2, 6:36 am)
Re: MMIO and gcc re-ordering issue, Benjamin Herrenschmidt, (Mon Jun 2, 5:53 pm)
MMIO and gcc re-ordering (Was: [PATCH] [POWERPC] Improve (in..., Benjamin Herrenschmidt, (Fri May 23, 8:36 am)
Re: MMIO and gcc re-ordering (Was: [PATCH] [POWERPC] Improve..., Benjamin Herrenschmidt, (Fri May 23, 8:50 am)
Re: MMIO and gcc re-ordering (Was: [PATCH] [POWERPC] Improve..., Benjamin Herrenschmidt, (Fri May 23, 6:47 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Benjamin Herrenschmidt, (Tue May 20, 6:02 pm)
Re: [PATCH] [POWERPC] Improve (in|out)_beXX() asm code, Trent Piepho, (Tue May 20, 6:21 pm)