login
Search
Robert Watson posted a very informative announcement of 'ethercons', an ethernet console driver for FreeBSD -current. He was inspired by a FREENIX paper about a Linux ethernet console driver, though has evidently improved upon this design by supporting both input and output, while the Linux driver provides only output monitoring. Robert describes the effort:
"As with the Linux driver, communication happens at the ethernet link layer, using protocol number 0x0666 (entertaining choice). The contents consist of a little meta-data (not found in Linux), and a nul-terminated string, although the kernel code currently generates only single characters due to the nature of the console code. ethercons implements both a low level putc() console interface, and a high-level pseudo-tty appropriate for /dev/console redirection and getty/login. Unlike the other low-level console drivers, ethercons does not implement low-level input checkc/getc, as ethercons is interrupt driven, and that interface is a polled interface that conflicts with the tty code."
The ethernet console driver currently does not support remote debugging due to using the ethernet layer of the network stack. Robert explains, "Since the debugger suspends scheduling, this means interrupt threads, netisrs, etc, aren't running, so for now, ethercons is disabled when 'in the debugger'." However, he does goes on to explain how this could be implemented, using Darwin's ethernet debugger as an example. Read on for the full discussion.
From: Robert Watson [email blocked]
To: current
Subject: ethercons: ethernet console driver for 5-current
Date: Mon, 20 Oct 2003 12:13:27 -0400 (EDT)
I had a fair amount of time over the last week running in disconnected
operation, and realized I had too many cables under my desk, so I spent a
bit of time exploring the FreeBSD console code. After reading a FREENIX
paper this summer on a Linux ethernet console driver, I took a pass at
implementing ethernet console support for FreeBSD. This driver is similar
to the Linux driver, although not binary-compatible on the wire, as this
driver supports both input and output, while the Linux driver supports
only output monitoring (and the protocol can't represent bi-directional
communication well). There are some potential work-arounds for this,
which I'll explore at some point. In general, the wire protocol is
probably the weakest part of the endeavor, but I'm having trouble finding
documentation for a decent wire console protocol that doesn't come with an
entire network stack attached.
As with the Linux driver, communication happens at the ethernet link
layer, using protocol number 0x0666 (entertaining choice). The contents
consist of a little meta-data (not found in Linux), and a nul-terminated
string, although the kernel code currently generates only single
characters due to the nature of the console code. ethercons implements
both a low level putc() console interface, and a high-level pseudo-tty
appropriate for /dev/console redirection and getty/login. Unlike the
other low-level console drivers, ethercons does not implement low-level
input checkc/getc, as ethercons is interrupt driven, and that interface is
a polled interface that conflicts with the tty code. I'm considering
adopting a timeout-driven model as done in ofw_console, but haven't
convinced myself that is entirely desirable. In addition, the ethercons
device is not available for I/O when in the debugger context, due to its
use of the network stack. To support this, I recently added a flags field
to the console definition, and a NODEBUGGER flag.
To enable support for ethercons, add "options ETHERCONS" to your kernel
configuration. A series of tunables and sysctls is available to tune the
behavior of ethercons:
kern.ethercons.ifnet_raise "ifconfig up" the interface prior to
reaching init so that ethercons may be
used in single usermode. Otherwise,
ethercons only becomes available when the
interface is brought up later by
dhclient/ifconfig/... Alternatively, for
network booted environments, the interface
may already be up.
kern.ethercons.interface_preference Interface name preference, if any.
Otherwise, the default is the first
ethernet interface. The most
recently used interface is
available read-only via
kern.ethercons.interface.
kern.ethercons.target Target ethernet address for the console target.
Otherwise, the default is ff:ff:ff:ff:ff:ff.
The ethercons client uses bpf; it's a fairly limited tool in its current
form. It has several modes of operation:
log Follow the console output of all ethernet consoles,
logging the output to various log files, named by the
source ethernet address of the messages (specified by
interface).
miniterm A minimalist interactive terminal program to be pointed
at a specific ethernet interface and hardware address.
send Send a string to a remote console as input (specified
by interface, target address).
sendcr Send a string to the remote console as input, along
with a carriage return (specified by interface, target
address).
tail Follow the console output of a particular ethernet console
(specified by interface and source address).
tailall Follow the console output of all ethernet consoles, even
though the results are potentially messy (specified by
interface).
It should be possible to create a more complete client for easier
interactive use; alternatively, the firewire console code binds a socket
for use with a telnet client, which could be done for ethercons, which
might be a better approach than writing more interactive code of that
sort.
You can set up a getty/login session on /dev/ethercons using /etc/ttys:
ethercons "/usr/libexec/getty Pc" xterm on secure
The changes consist of three parts: two new kernel files
(src/sys/dev/{ethercons.c, ethercons.h}), a kernel patch (ethercons.diff),
and a userland tool for monitoring/logging/communicating with the ethernet
console (src/usr.sbin/ethercons/{ethercons.c,ethercons.h}). Note that to
build the userland ethercons tool, the updated ethernet.h must be in
/usr/include/net due to its reliance on the new constant.
http://www.watson.org/~robert/freebsd/ethercons-0.2.tgz
I believe I have merged all the necessary patches to support ethercons
into the base tree, wich the exception of the tweaks to net/ethernet.h to
allocate a new ethernet protocol number.
I have no immediate plans to commit this code, but would appreciate
feedback. It appears to work fairly well for me, but it highlighted some
limitations to the FreeBSD networking and console code:
(1) Unless you use netgraph, it is "hard" to extended the set of ethernet
protocol handlers. Introducing dynamicism in the protocol switches
will introduce additional locking costs.
(2) Two levels of support are present in the console driver: first,
low-level console support provides a simple getc/checkc/putc interface
for kernel output and input. However, to support user process
interaction with the console, such as syslogd writing to the console,
or single-user mode, you need a tty. The two interfaces are fairly
incompatible due to the polled vs. interrupt-driven model. To
support the polled interface, the console driver needs to own the
buffer memory. To support the tty interface, the tty driver needs to
own the memory. The open firmware console works around this by
having timeouts that transfer data from a console-owned buffer to a
tty-owned buffer at intervals. The low-level console code races with
the timeout to "get there first" and extract data for the low-level
console when it is required. All in all, this puts console/tty driver
writers in a sticky situation if the underlying medium doesn't support
polled access naturally, and behaves in an interrupt-driven fashion.
So, the quick summary: it appears to work, it has limitations, and the
wire protocol sucks. I'd like to find a better wire protocol (without too
much overhead), and figure out a better answer for low-level console
interactions. The client is also pretty limited, and could do with some
improvement.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[email blocked] Network Associates Laboratories
From: Steve Kargl [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Mon, 20 Oct 2003 10:03:52 -0700
On Mon, Oct 20, 2003 at 12:13:27PM -0400, Robert Watson wrote:
>
> I had a fair amount of time over the last week running in disconnected
> operation, and realized I had too many cables under my desk, so I spent a
> bit of time exploring the FreeBSD console code. After reading a FREENIX
> paper this summer on a Linux ethernet console driver, I took a pass at
> implementing ethernet console support for FreeBSD.
Robert,
This looks very interesting! Can we run ddb over the
ethercon to debug a wedged machine?
--
Steve
From: Robert Watson [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Mon, 20 Oct 2003 13:41:05 -0400 (EDT)
On Mon, 20 Oct 2003, Steve Kargl wrote:
> On Mon, Oct 20, 2003 at 12:13:27PM -0400, Robert Watson wrote:
> >
> > I had a fair amount of time over the last week running in disconnected
> > operation, and realized I had too many cables under my desk, so I spent a
> > bit of time exploring the FreeBSD console code. After reading a FREENIX
> > paper this summer on a Linux ethernet console driver, I took a pass at
> > implementing ethernet console support for FreeBSD.
>
> This looks very interesting! Can we run ddb over the ethercon to debug
> a wedged machine?
Not currently. In the current implementation, the ethernet console picks
up its input, and generates output, using the ethernet layer of the
network stack. Since the debugger suspends scheduling, this means
interrupt threads, netisrs, etc, aren't running, so for now, ethercons is
disabled when "in the debugger" (db_active != 0). This permits other
console devices, such as serial console, to be used for the debugger,
however, at the same time.
To support ethernet debugging, the debugger would need to be able to drive
polling of the network interface in an interrupt-thread-free environment,
and reproduce more of the lower level network code (i.e., not use mbufs,
etc). This is feasible to do, but would probably require adding new
interfaces to the ethernet driver, and supporting only ethernet cards that
had these additional debugging interfaces. Compared to serial console,
you'd also have a lot more situations where the driver/hardware state
would be sufficiently inconsistent as to make debugging network-related
crashes difficult. On the other hand, Darwin runs quite well with a
network debugger; I believe they have a fairly complex UDP/IP
implementation in the network debugger, although I haven't inspected it.
Apple has the advantage, though, of providing very few ethernet drivers.
So I'm happy to look at it, but the level of time investment to get to
network debugging from the current (and pretty simple) ethercons device
will be fairly high. I know Jonathan Lemon was looking at network console
and debugging code previously, but I don't have copies of his patches. If
I had to guess, I'd assume he had modified the if_fxp driver, and perhaps
others, to provide an appropriate polled interface for use with a
debugger, but I don't know for sure. If someone has copies of these
patches, I'd be happy to take a look at them.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[email blocked] Network Associates Laboratories
From: Terry Lambert [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Tue, 21 Oct 2003 00:18:02 -0700
Robert Watson wrote:
> On Mon, 20 Oct 2003, Steve Kargl wrote:
> > This looks very interesting! Can we run ddb over the ethercon to debug
> > a wedged machine?
[ ... ]
> To support ethernet debugging, the debugger would need to be able to drive
> polling of the network interface in an interrupt-thread-free environment,
> and reproduce more of the lower level network code (i.e., not use mbufs,
> etc). This is feasible to do, but would probably require adding new
> interfaces to the ethernet driver, and supporting only ethernet cards that
> had these additional debugging interfaces. Compared to serial console,
> you'd also have a lot more situations where the driver/hardware state
> would be sufficiently inconsistent as to make debugging network-related
> crashes difficult. On the other hand, Darwin runs quite well with a
> network debugger; I believe they have a fairly complex UDP/IP
> implementation in the network debugger, although I haven't inspected it.
> Apple has the advantage, though, of providing very few ethernet drivers.
It's pretty nifty. It's very infrequent that a hang results in
something that can't be debugged remotely. The only cases are
when an IP address and default route are unknown to the machine
at the time of the crash. This can be overridden by providing
Open Firmware values, up front, to avoid the need to wait until
DHCP is up enough to get the IP (the DHCP implementation is very
low level, and happens very early on). Assuming the bug reproduces.
You're right about the card support: it's a polled mode driver;
on the other hand, it's not really that difficult, even though
many ethernet drivers are loaded into the kernel as kext's (Kernel
Extensions); FreeBSD boot code can do similar magic up front, in
loading drivers as kernel modules. So this actually should not be
very hard to put into any/all drivers. It could go in slowly, like
multicast, soft interrupt coelescing, or DEVICE_POLLING did, so it's
not that big an issue (IMO): at least it's not all or nothing.
> So I'm happy to look at it, but the level of time investment to get to
> network debugging from the current (and pretty simple) ethercons device
> will be fairly high. I know Jonathan Lemon was looking at network console
> and debugging code previously, but I don't have copies of his patches. If
> I had to guess, I'd assume he had modified the if_fxp driver, and perhaps
> others, to provide an appropriate polled interface for use with a
> debugger, but I don't know for sure. If someone has copies of these
> patches, I'd be happy to take a look at them.
The Darwin code is available, and isn't much changed in that area
from Jaguar to Panther, so the sources should still be good (FWIW),
if you wanted to look there. If anything gets done, I'd dearly
love it to get done so that there was binary compatability enough
to be able to cross-debug systems, but of course, that would be up
to whoever had the time to invest in doing the work.
A neat feature of Panther is the ability to actually do system
dumps over the wire, if you enable them (and set the magic values).
The graduate student who did the original network kernel debugging
code in Darwin did that work over the past summer (I did the code
for integrating the sysctl's for him). This might be more valuable
to FreeBSD than actually running a source debugger (or not), but
it's a lot of fun to play with (it uses a dumping protocol based on
a modified TFTP). It would at least be useful in shops with 10
or more BSD working machines and one monitoring box, which have the
occasional crash.
-- Terry
From: Robert Watson [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Tue, 21 Oct 2003 11:34:13 -0400 (EDT)
FYI, for those who expressed interest, the URL has changed slightly, and
I've posted some minor bugfixes and updates:
http://www.watson.org/~robert/freebsd/ethercons/
I'm also investigating the MOPRC protocol, and alternative ethernet
address choices. I did take a look at the Panasas ipgdb patches for
FreeBSD, but it looks like they wouldn't apply well in 5.x without
substantial modification (and a bit of evolution of our MP-handling in
device drivers).
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[email blocked] Network Associates Laboratories
From: Peter Jeremy [email blocked]
To: Robert Watson [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Wed, 22 Oct 2003 06:26:18 +1000
On Mon, Oct 20, 2003 at 12:13:27PM -0400, Robert Watson wrote:
> After reading a FREENIX
>paper this summer on a Linux ethernet console driver, I took a pass at
>implementing ethernet console support for FreeBSD.
A very worthy cause. I'm sure this has come up before but I think
you're the first person to produce actual code. Something that you
don't seem to address is security - which surprised me.
> This driver is similar
>to the Linux driver, although not binary-compatible on the wire,
>...
>As with the Linux driver, communication happens at the ethernet link
>layer, using protocol number 0x0666 (entertaining choice).
If Linux is using 0x0666, we should probably pick a different number
since we're not wire compatible. Though coming up with a common
protocol would be even better.
> In general, the wire protocol is
>probably the weakest part of the endeavor, but I'm having trouble finding
>documentation for a decent wire console protocol that doesn't come with an
>entire network stack attached.
MOP (as you point out later) or LAT have the advantage of being more
standard, but I'm not sure how well documented they are.
> A series of tunables and sysctls is available to tune the
>behavior of ethercons:
>kern.ethercons.ifnet_raise
>kern.ethercons.interface_preference
>kern.ethercons.target
Is there any way to specify ifconfig options? "media" and "mediaopts"
in particular may need to be specified to get the interface to talk to
the associated switch.
I presume kern.ethercons.target only specifies the MAC address
inserted into transmitted packets. Is there any way to restrict the
src address(es) of received packets? Does ethercons have any concept
of a current session or will it accept incoming packets from anywhere
at any time? The latter case would seem undesirable as (IMHO) it
makes it too easy to accidently send a command to the wrong system
Peter
From: Robert Watson [email blocked]
Subject: Re: ethercons: ethernet console driver for 5-current
Date: Tue, 21 Oct 2003 19:34:37 -0400 (EDT)
On Wed, 22 Oct 2003, Peter Jeremy wrote:
> On Mon, Oct 20, 2003 at 12:13:27PM -0400, Robert Watson wrote:
> > After reading a FREENIX
> >paper this summer on a Linux ethernet console driver, I took a pass at
> >implementing ethernet console support for FreeBSD.
>
> A very worthy cause. I'm sure this has come up before but I think
> you're the first person to produce actual code. Something that you
> don't seem to address is security - which surprised me.
Well, in a sense I did: the environment I use the ethernet console in is
one of trust of the link layer. I use DHCP, PXE, and NFS roots on the
boxes in question. That said, I should have more carefully documented the
security assumptions: that there will be no malicious attackers on the
wire, and because it's a link layer protocol, there will be no routing of
the packets in question from untrusted segments.
I did look at introducing cryptographic protections, but there are a
number of problems with this:
(1) I wanted to minimize the amount of transient protocol state. The
current implementation does not make any reliable transmission
guarantees. In order to prevent replay attacks, we would need to rely
on reliable timestamps or sequence numbers. In order to establish
sequence number legitimacy, we'd need to have a protocol component
that establishes freshness.
(2) It introduces the issue of key management. In my environment, since
the system configuration is shipped in the clear anyway, the keying
material would also get transfered in the clear, which would make the
whole arrangement somewhat moot.
The minimalist solution would introduce just keyed integrity, perhaps
using HMAC-MD5 and a shared secret. The problem does come back to
sequencing: if you assume the LAN is subject to replay and monitoring,
timestamps aren't sufficient, you actually need sequencing and freshness.
This means doing a Needham-Schroeder shared key exchange or a related
notion to get a session key. This requires the client and server to
detect and handle reboots, for example. I am willing to do the
implementation, but without a wire protocol worth committing to yet, I was
unwilling to invest the time.
> > This driver is similar >to the Linux driver, although not
> binary-compatible on the wire, >... >As with the Linux driver,
> communication happens at the ethernet link >layer, using protocol number
> 0x0666 (entertaining choice).
>
> If Linux is using 0x0666, we should probably pick a different number
> since we're not wire compatible. Though coming up with a common
> protocol would be even better.
The problem with the Linux protocol was that it had no concept of
direction. I'm still hopeful I'll find a better wire protocol to model
the implementation on.
> > In general, the wire protocol is
> >probably the weakest part of the endeavor, but I'm having trouble finding
> >documentation for a decent wire console protocol that doesn't come with an
> >entire network stack attached.
>
> MOP (as you point out later) or LAT have the advantage of being more
> standard, but I'm not sure how well documented they are.
I have found some documentation of MOP and LAT, although it's not clear to
me there's enough information to do an independent implementation without
referencing an existing one. There is a Linux implementation, so I'll
take a look at that. One of my concerns here is that it could be these
protocols require a fairly deep DECnet stack, and I'm not all that
interested in building big protocol stacks. However, until I'm done with
the reading, I can't commit myself either way :-).
> > A series of tunables and sysctls is available to tune the
> >behavior of ethercons:
> >kern.ethercons.ifnet_raise
> >kern.ethercons.interface_preference
> >kern.ethercons.target
>
> Is there any way to specify ifconfig options? "media" and "mediaopts"
> in particular may need to be specified to get the interface to talk to
> the associated switch.
Currently, no. If we do want to add this sort of infrastructure, we might
want to provide common infrastructure for both the diskless booting
environment, which also attempts to raise the interface before getting to
the userland configuration.
> I presume kern.ethercons.target only specifies the MAC address inserted
> into transmitted packets. Is there any way to restrict the src
> address(es) of received packets? Does ethercons have any concept of a
> current session or will it accept incoming packets from anywhere at any
> time? The latter case would seem undesirable as (IMHO) it makes it too
> easy to accidently send a command to the wrong system
Right now, ethercons defaults to a broadcast for outgoing, and accepts any
locally destined packet on the inbound. There is no notion of session.
This is both an upside and a downside: I've actually taken advantage of
this behavior to monitor using a common monitoring host, and then used an
interactive session simultaenously from another host. However, my hosts
all have fairly different ethernet addresses, making the chance of an
accidental collisions and "sending a command to the wrong machine" low. It
would be quite easy to add another configurable entry to say "only accept
input from address <x>", however. Per my comments earlier in this e-mail,
my goal in the current implementation has been to avoid the complexity of
higher levels of the stack: complex addressing, session management, etc.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[email blocked] Network Associates Laboratories
