I'm in the middle of implementing a userspace client for usbip and I
strongly feel that the protocol needs to be changed before it is merged.

 - I'm unconvinced that TCP is the correct protocol to be running this over.
   I understand the reluctance to use UDP, but the protocol is fundamentally
   packet-based.  If TCP is used, the delimitation of packets within the
   stream needs to be much more robust.  I've managed to wedge the VHCI driver
   a number of times in ways that just wouldn't be possible if we were using
   a packet protocol instead of a stream protocol.
 - Endianness.  This is a mess.  The usbip protocol is big-endian, but the
   encapsulated usb protocol is little-endian.  This doesn't matter to the
   people who are just tunnelling usb from one computer to another, but for
   someone implementing a usbip client, it's very confusing.
 - The protocol needs an officially assigned port number.  Port 3240 is 
   already assigned to Tony Matthews <tmatthews&triomotion.com> February
   2002 (see http://www.iana.org/assignments/port-numbers)
 - There are actually two completely different protocols in use.  First,
   the usbipd daemon listens on port 3240, and handles device discovery.
   When usbip successfully attaches to usbipd, both sides of the connection
   pass the socket fd into the kernel and the protocol changes.
 - The protocol sends a 48-byte packet header for every command (and every
   response).  It's cunningly hidden as a union.
   
I think the protocol would be immeasurably improved by going through the
IETF RFC process and getting feedback from networking experts.  Failing
that, I have some suggestions about how to improve it.  I was hoping to
get my client finished before I started mucking with the protocol though.

(I have some other comments on the implementation, but they're a separate
issue).

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but ...
[ message continues ]

If you want reliable transport with record boundaries an alternative
would be also SCTP. Main drawback is that firewalls often don't support it
though (but presumably that wouldn't be a big issue for this)

-Andi
-- 
ak@linux.intel.com
--

[ message continues ]

I would also have proposed SCTP. The telco carries seems be very happy
with it.

Regards

Marcel


--

[ message continues ]

I'm actually looking into the sunrpc protocol.  That has the advantages:
 - Already has an in-kernel implementation
 - Widely understood and properly documented
 - Can be run over TCP or UDP or even RDMA

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
--

[ message continues ]

and :
   - already supported by most firewalls.

Willy

--

[ message continues ]

Then just document it, no big deal.
Yeah, the current code isn't the cleanest here (sparse throws up some
warnings), but it's not that much work to fix it up, it's on my todo



Why mess with the RFC process, is that really necessary for something
like this?

Windows has had this for years, no need for a RFC there, and if we just
document this well, no need for one here either.

thanks,

greg k-h
--

[ message continues ]

Erm, did you not read what I wrote?  USB is packet based.  TCP isn't.

I'm not talking about the code.  I'm talking about the protocol.  It's a

Yes, it really is.  It complicates the protocol, complicates the
implementation, introduces unnecessary state, and makes it impossible to

It helps clarify the odd corners of any protocol.  I don't have the
impression that it's a terribly heavy-weight process -- though we can

Yes, and as a result we can't interoperate with Windows.

By the way, is this actually built into Windows or just available as
several mutually incompatible and pay-for products?  I did some
searching a few months ago and didn't come up with anything official
from Microsoft.

Even if we don't go through the RFC process, just writing down the
on-wire protocol should be mandatory for taking this kind of thing into
the kernel.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
--

[ message continues ]

Sorry, early morning, no coffee yet...

I think in the end, we should still use TCP otherwise you just end up




There is nothing official, there are various incompatible and pay-for

Why, isn't the actual implementation better than a document?  :)

thanks,

greg k-h
--

[ message continues ]

Which brings us to the alternate -- that we need better framing in the


Patches don't seem appropriate for a design discussion.  I'm more than
happy to make suggestions about how to unify the two protocols.  I'll

Surely you know that writing things down forces you to understand it
better?

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
--

[ message continues ]

It's an enormous help for anyone working on the code, either now or
in the future.  Reverse engineering a protocol from source code is
difficult.

Dave



*************************************************************************************************************************************************************************************************************************************************

NICE CTI Systems UK Limited ("NICE") is registered in England under company number, 3403044.  The registered office of NICE is at Tollbar Way, Hedge End, Southampton, Hampshire SO30 2ZP.

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.

Monitoring: NICE may monitor incoming and outgoing e-mails.

Viruses:  Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.

****************************************************************************************************************************************************************************************************************************************************

 

--

[ message continues ]

Here's a document for discussion.  No code yet, though I'm quite willing
to modify the current usbip code to follow this new protocol.  Just a
matter of time.

http://www.kernel.org/pub/linux/kernel/people/willy/usbip/usbip-protocol-draft-1

To save some time for reviewers, here's a list of decisions I took while
working on the document.  I appreciate that some of the decisions I made
were not necessarily those another designer might have made, so I ask
that any comments along the lines of "I would have done it differently"
include a really good reason.

----

I've merged the two protocols.  It is no longer the case that the protocol
completely changes when userspace drops the connection into the kernel.
It's still possible to do a hybrid implementation where userspace
implements version, list and claim and kernelspace handles release,
submit and unlink.

During development of my own implementation, I noticed the client and
server get confused about where the packets were in the TCP stream.
No more; if using a stream protocol, we wrap each packet in a 'RM'.

I considered using the whole of sunrpc.  The encoding is quite
heavyweight and I felt I could do better by specialising the protocol to
USB's purposes.

'get version' is now a call rather than including a version field in
every request.

Devices are now referred to as an ascii string rather than an encoded
4-byte quantity.  This helps userspace configure the device and lets us
interoperate with other OSes that might want to implement this protocol.

Instead of transmitting the device number in every command, we now bind
each socket to a particular device.  This was already what happened,
so it was just overhead.

I've split 'cmd_submit' into four commands (control, data, isoc, irq).
That gives us the ability to make 'data' very small.

There's no need to respond with the call number to each call -- the
caller should be using the call identifier to find out what type of
call it was.  Often the implementation ...
[ message continues ]

Have you given any thought to forward compatibility?  I imagine it's 

Is this string intended to be meaningful to a human?  Or could it be 

A number of important commands are missing.  Clear-Halt springs to

This isn't mentioned in your document.  There are "claim device" and 
"release device" calls, but it doesn't say anywhere that only one 
device can be claimed at a time.  Nor is there a status code for "No 

What happens when your protocol is used with a non-reliable transport?  

The reply to a submit call will be sent when the USB transfer is
complete.  Suppose the transfer takes a long time.  How does the client
tell the difference between a long-running transfer and an unreceived
submit?  Do you essentially assume that all protocol transfers are
reliable?

There isn't any field in the submit reply to report the status of the 
transfer (as opposed to the status of the submission).  How do errors 

What about the extra flags that go with URB submissions?  
URB_NO_INTERRUPT and URB_SHORT_NOT_OK might well be useful, 
URB_ZERO_PACKET is certain to be needed, and even URB_ISO_ASAP might 

It will not be possible to support all the features of Linux's USB 
stack for these types of transfers (I'm referring to the way the 
bandwidth isn't released if a new URB is submitted during the 
completion callback).

Both types will require an additional "interval" field for submission,
and Isochronous will require "start_frame", "number_of_packets", and a
list of packet descriptors as well.  It also will require a status and
length for each component packet in the reply.

In the unlink call, what is the "seqnum" field?  Is it supposed to be 
the call identifier of the corresponding submit call?

Alan Stern

--

[ message continues ]

By forward compatibility, you mean when we rev the protocol to version 2
and try to communicate?  My thought was:

 - v1 host, v1 device -- status quo
 - v1 host, v2 device -- host will simply disconnect
 - v2 host, v1 device -- host can choose to speak v1 protocol or
   disconnect
 - v2 host, v2 device -- v2 protocol will be spoken

If we have the host transmit the highest version protocol it supports,
the device could reply with the highest version it supports.  Or we
could have the host transmit all the protocol versions it's willing to
support (eg 1, 3, 4 because we found a horrible security problem in

It needs to be typable by a human.  It's up to the implementation
whether it wants to report "1-1" so that "usbip --attach localhost 1-1"
continues to work.  I don't mind if I have to type "usbip --attach
localhost Jhd%s" as long as that's one of the options I get from usbip
--list.  It's not intended to be "usbip --attach 'AuthenTec, Inc. AES2501
Fingerprint Sensor'" (though I suppose a sufficiently imaginative
implementation could take any arbitrary string and try to DWIM ... that
wouldn't be prohibited).  Refusing to accept a string that came

Set Configuration and Set Interface are requests on the command pipe
according to the usb 2.0 spec (these would be a control message
targetted at endpoint 0).  I don't find Reset Device in usb 2.0; in
usb-storage there's Bulk-Only Mass Storage Reset which is again a
request on the command pipe.  I also don't find Clear Halt in usb 2.0;
but if I'm reading 9.4.5 correctly, the Halt state is cleared by sending

This is true.  I'll make that more clear.  In fact, drawing a state

Good question, and one I hadn't been thinking about too hard since the

How about this:

 - The sender has a timeout after which time, it will re-send the call.
 - If the recipient receives a call that is still outstanding, it
   replies with a status akin to NFS's EJUKEBOX that means "It's in
   progress".
 - If the recipient receives a call for a ...
[ message continues ]

It's referred to as SetPortFeature(PORT_RESET) in section 11.24.2.13;  

In fact it's cleared by sending ClearFeature(ENDPOINT_HALT) to the 
appropriate endpoint.

Never mind the details; the important point is this: All these 
requests, in addition to sending data over the USB bus, have to modify 
the host's internal state.

	Set-Configuration updates the list of available interfaces
	and endpoints.

	Set-Interface updates the list of available endpoints.

	Reset-Device requires the host to verify that the device is
	still connected to the port and its descriptors haven't
	changed.  The host also has to restore the former configuration
	and altsettings.

	Clear-Halt requires the endpoint's data toggle value in the 
	host controller to be reset.

So it isn't good enough for the client simply to send a few packets; 
the server needs to know when these events happen.  In theory they can 
be detected by parsing packets as they are sent, but IMO it would be 


Should the server send a "submission accepted" message some time before
the first timeout occurs?  For long-running requests this would

This requires the server to keep each completed reply for some time.  

In brief, no.  The caller needs to know; retrying isn't always the 

They do make sense at this level, just as they make sense in usbfs.
(However the point of USB_SHORT_NOT_OK would be lost; the semantics of 
that flag would need to be extended for it to be useful.)

Alan Stern

--

[ message continues ]

Mmm.  In this case, we don't have an upstream hub.  The vhci-hcd driver
is on the host and sends URBs across the network where they're received
by the stub driver.  The stub then sends them to the exported device.

I don't have a USB hub, so I can't tell you what happens if the exported
device is plugged into one.

What causes Linux to send a 'reset' to a hub?  What do we do in similar
circumstances if the device isn't attached to a hub?  Is there a *hci
specific way of resetting a bus that's directly attached?

(I suspect you're going to be completely right here and we need to add a
'device reset' message to the protocol.  What isn't clear to me is how
the stub on the device side is going to communicate that reset to the

I see these requests need to be understood by the vhci (host) side, and
by the device itself, but it's not clear to me that they need to be
comprehended by the stub driver on the device side.  OTOH, I'm a long
way from being an expert on the USB protocol, so please tell me again


That's an implementation not protocol question ... I suspect that

I think that would be a reasonable thing to do.  Of course, the message
could be dropped on the way, so device-side implementations MAY do this,

When the device-side receives another packet to the same endpoint,
perhaps?  Does USB permit multiple outstanding commands to the same

OK, but if the transfer didn't complete, the submission doesn't have a
status either, so we can use the same status field for both protocol

OK.  I think I'm going to change the 4 byte call id to be a 1 byte
call id, and 3 bytes of padding for non-submit commands.  For submit
commands, I'll use 1 byte for the direction + endpoint.  That leaves 2
bytes for flags.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
--

[ message continues ]

I strongly suggest, if you are going to be touching USB code, that you
spend the $10 and get yourself a hub, I'm sure Intel might even allow
you to expense it.

If not, I'll take up a collection for you at the Plumbers conference :)

thanks,

greg k-h
--

[ message continues ]

I'm sorry I'm barging into the conversation, but a HC without a
root hub is rather shocking. One would think it easier to emulate
a root hub with one port in HCD than to teach the whole stack about
such thing as a hubless HC.

-- Pete
--

[ message continues ]

The way I've always understood it, there is, by definition,
always an upstream hub. It's the entity controlling the port
to which the device is connected. In the case discussed here,
I guess the stub would have to play that part.

Am I wrong?

--=20
Tilman Schmidt                    E-Mail: tilman@imap.cc
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Unge=F6ffnet mindestens haltbar bis: (siehe R=FCckseite)

[ message continues ]

Sometimes I find the terminology a little confusing.  "Client" and
"server" are clear enough, as is "device-side" (synonymous with
"server").  I guess vhci-hcd runs on the client, right?

But "host" is ambiguous; vhci-hcd is a host controller driver on the
client and [uoe]hci-hcd is a host controller driver on the server.  
Thus both sides are hosts.  Also, "stub" isn't too clear.  Is vhci-hcd
a stub driver?  Or does the stub driver run strictly on the server?

Getting back to your question...  All devices have an upstream hub,
except for root hubs.  In this case the device's upstream hub is the
physical hub it is plugged into if it plugged into a hub, or the
physical root hub it is plugged into otherwise.  Either way, the
upstream hub has to be managed by the server, not by the client.  
That's another reason why device reset needs to be treated as a special
call.

Similarly, suspend and resume involve sending requests to the upstream 

It shouldn't make any difference.  If it does then there's something 

I'm not sure I understand the question.  Linux sends a port-reset 
request whenever anyone calls usb_reset_device().  Is that what you 

Every device is attached to a hub, except for root hubs.  
usb_reset_device() fails if its argument is a root hub.  (Root hubs 

USB doesn't reset buses; it resets devices (or more precisely, a 
device's port on its upstream hub).  So I guess you're asking if there 
is a driver-specific way of resetting a port on a root hub.  The answer 


Okay.  You're wrong.  :-)

The server's USB stack needs to know about these requests because they
need to affect hardware and software state on the server.  If you tried
to pass one these things directly from the client to the device, you'd
find that suddenly things weren't working.  The hardware and software
on the server would no longer be able to communicate with the device

This isn't a hardware-level issue; it's a software issue.  Linux's USB 
stack allows multiple outstanding requests to ...
[ message continues ]

[Sorry, I meant to send this mail before I left on Friday ... best laid
plans, etc]


'Client' and 'Server' are totally not clear.  Is the 'server' the big
machine in the closet without USB devices, or is the machine with the
device attached to it 'serving' the device to the machine without usb
devices?

(I'd like to give a big shout-out to my homies working on X Windows for
making this confusion possible)

Controller machine:

Loads vhci-hcd
runs usbip --attach machine-b 2-2

Target machine:

Loads usbip
runs usbipd
runs bind_driver --usbip 2-2

The controller machine now gets a device 9-2 which accesses the target
machine's 2-2 device.

I suppose it's my fault for not documenting this clearly in the protocol

The usbip module is also referred to as the stub driver:


OK, I was confused, I didn't realise the *hci included a hub.  I always
thought of a hub as an external device.

So the hub in the usbip scenario is on the controller machine.  Obviously,
the target machine also has a hub (ohci/uhci/ehci), but from the point
of view of sending a reset, it would be sent to vhci which then has to
communicate it to the device somehow.  So yep, let's include a reset
message in the protocol and that's how it will work.





I'm going to assume you mean 'device server' in this context (now
referred to as target machine).

I just don't understand why the target machine's stack needs to know
which endpoints are available.  Is it for when the controller detaches?
I would think that a USB reset should be issued to the device when
control changes between a real driver (eg usb-storage) and the stub

I guess the question is where the queueing happens -- does it happen in
the *hci driver, or does it happen in the device?  Either way, we could
choose whether to queue the urbs on the controller machine or on the
target machine -- it's a complexity / performance tradeoff.  I don't

Works for me.  We'll still want to document what they are since we
presumably want to ...
[ message continues ]

Hi all,

I have been looking at the USBIP stuff and don't understand the
generic probe issues.

In your second patch:
static struct usb_device_id stub_table[] = {
+#if 0
+	/* just an example */
+	{ USB_DEVICE(0x05ac, 0x0301) },   /* Mac 1 button mouse */
+	{ USB_DEVICE(0x0430, 0x0009) },   /* Plat Home Keyboard */
+	{ USB_DEVICE(0x059b, 0x0001) },   /* Iomega USB Zip 100 */
+	{ USB_DEVICE(0x04b3, 0x4427) },   /* IBM USB CD-ROM */
+	{ USB_DEVICE(0x05a9, 0xa511) },   /* LifeView USB cam */
+	{ USB_DEVICE(0x55aa, 0x0201) },   /* Imation card reader */
+	{ USB_DEVICE(0x046d, 0x0870) },   /* Qcam Express(QV-30) */
+	{ USB_DEVICE(0x04bb, 0x0101) },   /* IO-DATA HD 120GB */
+	{ USB_DEVICE(0x04bb, 0x0904) },   /* IO-DATA USB-ET/TX */
+	{ USB_DEVICE(0x04bb, 0x0201) },   /* IO-DATA USB-ET/TX */
+	{ USB_DEVICE(0x08bb, 0x2702) },   /* ONKYO USB Speaker */
+	{ USB_DEVICE(0x046d, 0x08b2) },   /* Logicool Qcam 4000 Pro */
+#endif
+	/* magic for wild card */
+	{ .driver_info = 1 },
+	{ 0, }                                     /* Terminating entry */
+};

You have changed your driver to a wild card for everything. Fine, it
makes sense, your probe routine should check if it wants a new device
and fail if it does not want it.

My problem is I am using the latest Suse enterprise linux which uses
kernel 2.6.18..... In that kernel the probes are done by the device
system and if a modprobe definition says the module is for this device
it gets loaded (by udev I think) and probed. fine. If the module is
already loaded the kernel probes the modules in the order they were
insmoded (if they say they want the particular VID/PID/CLASS/wildcard
etc.)

The problem is the probing is done by the kernel driver stuff and the
insmoding and user space intelligence is done by udev. Once a kernel
module is installed it must take all devices that match it's device ID
table.  Remember USBIP matches everything! That is because (at least
in the 2.6.18 kernel) the kernel driver stuff stops probing after ...
[ message continues ]

No it doesn't, it continues on.  Or it should.  Perhaps it's been fixed
since then, I would rely on that kernel to be looking like anything
"real" these days, it is very old and only supported by Novell, and not
the target audience for this patch at all.

thanks,

greg k-h
--

[ message continues ]

--

[ message continues ]

It is unfortunate that these two words have been used in multiple ways, 
with meanings that are sometimes only subtly different and sometimes 
grossly conflicting.

Isn't it standard in networking circles for "server" to mean the system
running a daemon that waits for incoming connections and "client" to
mean the system running an on-demand program that initiates a

Okay, "controller" and "target".  I'm not sure that "controller" is the
best word either, since it might be confused with "USB host

Hmmm.  Above you said that the controller system runs a program named 
"usbip" and the target system loads a module named "usbip".  Since the 
stub driver is the usbip _module_, it must run on the target (AKA 

Yes; it is referred to as a "virtual root hub", or just "root hub" for 
short.  We didn't make this up; section 10.2.8 of the USB spec is 
entitled "Root Hub".

FYI, the *hci-hcd modules are collectively known as "host controller
drivers", or HCDs for short.  We USB guys sling that term around a lot


"Detaches" isn't a USB term.  Do you mean it in the sense of a SCSI 
initiator detaching from its target?

In any event, the answer is that each endpoint has several properties
associated with it (direction, type, maximum packet size, and so on),
and these properties have to be known by both the target's USB stack
and the target's USB host controller hardware.  The hardware uses DMA
to access various in-memory data structures describing the I/O
operations it should perform, and the format and contents of these data
structures depends on the endpoints' properties.  (To make matters
worse, the hardware will sometimes maintain an internal cache for parts
of the data structures and some of the properties; when the properties 
change the cache must be flushed.)

With new configurations or altsettings you end up with a new set of
endpoints, or the same old endpoints with a new set of properties.  

I suppose the controller system, as part of its normal initialization ...
[ message continues ]

You should post this in the message itself, I think it is shorter than


Both of those are necessary, so you are going to have to define them
(mice/keyboards use interrupt and audio uses iso, so it is very common,

What do you mean, you are just going to stick with BE?

thanks,

greg k-h
--

[ message continues ]

I don't think I have any usb audio devices.  Do you have any spares I

I'm just going to use BE in the protocol, even though it's encapsulating
a LE protocol.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
--

[ message continues ]

I only have one left, and I need it for testing, sorry.  There are quite
a few cheap usb audio headsets out there if you just want to buy your

That's fine with me.

thanks,

greg k-h
--

[ message continues ]