Re: USB transfer_buffer allocations on 64bit systems

From: Alan Stern
Date: Wednesday, April 7, 2010 - 12:13 pm

Linus specifically requested us to avoid using kzalloc in usbfs.  I
can't find the message in the email archives, but Greg KH should be
able to confirm it.

As long as we're imitating kmalloc for one use, we might as well make

Yes, the terminology got a little confused between the PCI and DMA
realms.  I agree, "coherent" is better.

BTW, although some EHCI controllers may support 64-bit DMA, the driver 
contains this:

	if (HCC_64BIT_ADDR(hcc_params)) {
		ehci_writel(ehci, 0, &ehci->regs->segment);
#if 0
// this is deeply broken on almost all architectures
		if (!dma_set_mask(hcd->self.controller, DMA_BIT_MASK(64)))
			ehci_info(ehci, "enabled 64bit DMA\n");
#endif
	}

I don't know if the comment is still true, but until the "#if 0" is 
removed, ehci-hcd won't make use of 64-bit DMA.

Note also that just before the lines above, ehci-hcd.c contains this
comment:

	 * pci_pool consistent memory always uses segment zero.

(Nowadays the driver uses dma_pool, not pci_pool.)  As far as I can
tell, this comment isn't true on 64-bit systems, but nevertheless,
ehci-hcd relies on it.  Of course it _is_ true for devices whose DMA
mask indicates they can't use memory above 4 GB -- but if ehci-hcd is
changed to enlarge the DMA mask then how do we force dma_pool memory to
lie below 4 GB?

[Actually it isn't _really_ necessary for the dma_pool to lie below 4 
GB.  But ehci-hcd allocates several pools, and it _is_ necessary for 
all of them to lie in the _same_ 4 GB segment.  The easiest way to do 
that is to put them all in segment 0.]

Alan Stern

--

From: Robert Hancock
Date: Wednesday, April 7, 2010 - 4:59 pm

The comment is wrong (or at least outdated or based on an incorrect 
assumption), but you're right, currently 64-bit DMA is not used on any 
EHCI controllers. It could be, but it sounded like the consensus was it 
wasn't worth the risk. Apparently Windows 7 started using it, and then 
had to put out a patch because some NVidia EHCI controllers indicated 
64-bit support but it didn't work properly. So you'd have to blacklist 
those controllers, at least.
--

From: Greg KH
Date: Wednesday, April 7, 2010 - 5:33 pm

I think someone tried to remove it recently, but I wouldn't let them :)

What a mess, hopefully xhci will just take over and save the world from
this whole thing...

thanks,

greg k-h
--

From: Robert Hancock
Date: Thursday, April 8, 2010 - 5:01 pm

True.. except for the fact that the xhci driver currently doesn't do 
64-bit DMA either, nor does it support MSI even though the HW supports 
it (surprisingly enough the NEC Windows driver does, MSI-X even). At 
this point only Intel likely knows how to do this properly, though, 
since AFAICS the spec isn't publicly available yet.
--

From: Sarah Sharp
Date: Friday, April 9, 2010 - 9:50 am

I hate to break it to you, but 64-bit DMA support is optional for an
xHCI implementation.  There's a bit in HCCPARAMS that tells whether the
host supports it (see the HCC_64BIT_ADDR macro in xhci.h).  The xHCI
driver currently doesn't do anything with that bit, although it should.

What makes you think that?  I've seen URB buffers with 64-bit DMA
addresses.  I can tell when the debug polling loop runs and I look at
the DMA addresses the xHCI driver is feeding to the hardware:

Dev 1 endpoint ring 0:
xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841

So the TRB at address 71a49800 is pointing to a buffer at address
0x0008000001000680.

If I'm setting a DMA mask wrong somewhere, or doing something else to

There's a patch from AMD to enable MSI-X.  The code was there, just

I have tried very hard to fix this, and will continue to do so.

Sarah Sharp
--

From: Robert Hancock
Date: Friday, April 9, 2010 - 4:38 pm

On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp

I'm not sure why the address would be that huge, unless it's not
actually a physical address, or there's some kind of remapping going

The DMA mask for the controller isn't being set anywhere (in the
version that's in Linus' current git anyway). In that case it'll
default to 32-bit and any DMA mappings above 4GB will need to be
remapped. The controller driver doesn't do the mapping itself but the
USB core does, passing in the controller device as the one doing the
DMA, so if the controller's DMA mask is set to 32-bit then the buffers
passed in will get remapped/bounced accordingly.

You should likely be setting the DMA mask for the controller device to
64-bit if the HCC_64BIT_ADDR flag is set, similar to the code in
ehci-hcd.c which is currently #if 0'ed out.

You can see the currently set mask in sysfs under

Ah, I see it. That could presumably be tested now (the NEC D720200F1
chip on the Asus U3S6 card I have seems to support MSI-X with 8
--

From: Robert Hancock
Date: Saturday, April 10, 2010 - 10:02 am

That is a bit suspicious, yes. With the DMA mask at default I don't
expect that should happen. Sarah, what kind of traffic was happening
when you saw that (bulk, isochronous, etc)?

If this apparent EHCI problem is reproducible, it might be useful to
add some code that warns if anything non-zero gets written to the
hw_bufp_hi and hw_buf_hi members in the descriptors.

Also, booting with mem=4096M on an affected system would also tell you
if it's related to using memory above 4GB.
--

From: Sarah Sharp
Date: Monday, April 12, 2010 - 11:56 am

Ring 0 is the default control ring, so it must be control transfers.
That's the first control transfer on the ring (and it didn't fill), so
it must have come from the USB core.

I was running the USB mass storage driver, and the bulk endpoint rings
don't have the high 32-bits of the address set.  It mostly uses the
scatter-gather interface, which calls usb_buffer_map_sg(), so that's not
surprising.

Sarah Sharp
--

From: Robert Hancock
Date: Monday, April 12, 2010 - 1:39 pm

On Mon, Apr 12, 2010 at 12:56 PM, Sarah Sharp

Is this machine using an IOMMU or something which would be remapping
physical addresses? That address 0x0008000001000680 seems huge, I
don't see how it could even be a valid bus address otherwise..
--

From: Sarah Sharp
Date: Monday, April 12, 2010 - 1:58 pm

Oh, shoot, nevermind.  That TRB has a slightly different format, where
the setup data for the control transfer is substituted for the DMA
buffer address.  I'll have to look through my logs to see if there's any
real 64-bit DMA addresses in there, and fix xHCI's DMA mask.

Sarah Sharp
--

From: Daniel Mack
Date: Monday, April 12, 2010 - 4:17 am

For more clearance what the functions actually do,

  usb_buffer_alloc() is renamed to usb_alloc_coherent()
  usb_buffer_free()  is renamed to usb_free_coherent()

They should only be used in code which really needs DMA coherency.

All call sites have been changed accordingly, except for staging
drivers.

Signed-off-by: Daniel Mack <daniel@caiaq.de>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: Greg KH <gregkh@suse.de>
Cc: Pedro Ribeiro <pedrib@gmail.com>
Cc: akpm@linux-foundation.org
Cc: linux-usb@vger.kernel.org
Cc: alsa-devel@alsa-project.org
---
 Documentation/DocBook/writing_usb_driver.tmpl  |    2 +-
 Documentation/usb/dma.txt                      |    4 +-
 drivers/hid/usbhid/hid-core.c                  |   16 +++++-----
 drivers/hid/usbhid/usbkbd.c                    |   12 ++++----
 drivers/hid/usbhid/usbmouse.c                  |    6 ++--
 drivers/input/joystick/xpad.c                  |   16 +++++-----
 drivers/input/misc/ati_remote.c                |   12 ++++----
 drivers/input/misc/ati_remote2.c               |    4 +-
 drivers/input/misc/cm109.c                     |   24 +++++++-------
 drivers/input/misc/keyspan_remote.c            |    6 ++--
 drivers/input/misc/powermate.c                 |   16 +++++-----
 drivers/input/misc/yealink.c                   |   24 +++++++-------
 drivers/input/mouse/appletouch.c               |   12 ++++----
 drivers/input/mouse/bcm5974.c                  |   24 +++++++-------
 drivers/input/tablet/acecad.c                  |    6 ++--
 drivers/input/tablet/aiptek.c                  |   14 ++++----
 drivers/input/tablet/gtco.c                    |   12 ++++----
 drivers/input/tablet/kbtab.c                   |    6 ++--
 drivers/input/tablet/wacom_sys.c               |   10 +++---
 drivers/input/touchscreen/usbtouchscreen.c     |    8 ++--
 drivers/media/dvb/dvb-usb/usb-urb.c            |    7 ++--
 drivers/media/dvb/ttusb-dec/ttusb_dec.c        |    6 ++--
 drivers/media/video/au0828/au0828-video.c      |    4 +-
 ...
From: Daniel Mack
Date: Tuesday, April 13, 2010 - 11:16 am

Is this ok? As it's quite big, I think it should be merged soon if there
are no objections.

Thanks,
--