KernelTrap interviewed Keith Owens, an experienced kernel hacker who has long contributed to the Linux kernel. His contributions include updating ksymoops and modutils, both of which he maintains. He also works on kbuild 2.5. Earlier, he built the original Integrated Kernel Debugging patch. He's also working on kdb and XFS.

---

Jeremy Andrews:
Please share a little about yourself and your background...

Keith Owens:
Born in Tenby, South Wales, 1952. BSc in Computer Studies at
Loughborough University. Emigrated to Australia in 1978. Married, two
kids (one of each), neither want to work in computers, although they
are quite happy to play games for hours on end and expect the in house
computer expert to fix all problem immediately ;).

JA:
How and when did you get started with Linux?

Keith Owens:
I was originally a mainframe systems programmer, on IBM, ICL, CDC etc.,
selling security software for IBM's MVS and Fujitsu's MSP. To do the
documentation and distribute the software (on 9 inch open reel tape!) I
needed a home machine. It was "obvious"[1] that Unix was the coming
thing so I got an NS32032 running an early version of SysV.

From there I moved to SCO Xenix but in both cases there was appalling
support from the vendors. Melbourne PC User Group introduced me to
Linux 0.99pl12, a week after installing Linux I blew away all remnants
of Xenix.

[1] "Obvious" in 1980, it has taken a while but we got there.

JA:
How did Linux 0.99pl12 compare to the other available offerings at that time?
You must have been impressed, to have been using it ever since. What about
Linux back then did you like so much?

Keith Owens:
Free source, free compilers, support it yourself. The vendors of the
closed source versions of Unix gave me the usual run around.

  Try this, try that, did you turn it on :(
  Nobody else has reported that problem, it must be your fault.
  Eventually - it is a bug, fixed in a later release which you have to pay for.

I would rather spend ten hours bug chasing myself than ten minutes
on the phone with "vendor support" which does not support you.

JA:
What are some of the contributions you've made?

Keith Owens:
In the 1.0 and 1.2 days I was involved with IP masquerading and some
network fixes. No traces are left now, that code has been rewritten
several times.

In 1998 I got fed up with the ksymoops code in the kernel tree, the
program that decoded kernel oops back to readable symbols. It was the
only kernel code that required C++, it only handled a couple of
architectures, it did not handle all messages and it did not handle
modules correctly. I became the ksymoops maintainer, rewrote it in C,
moved it outside the kernel, tried to handle all architectures, and
improved the modules support.

As part of the ksymoops changes, I did some patches to what was then
modutils 2.2 to improve oops reporting on modules. However modutils
was not being updated, the maintainer was too busy, so I volunteered to
maintain modutils as well.

I built the Integrated Kernel Debugging (IKD) patch by gathering lots
of kernel debugging patches that were floating around. There were
conflicts between the patches and some were poorly documented, IKD
pulled them together, cleaned up the conflicts and documented how to
use the tools to debug the kernel. Other people contributed IKD
patches, in particular Andrea Arcangeli and Mike Galbraith. I ran out
of time to maintain IKD so now Andrea and Mike look after it.

There are perennial problems with the kernel build for 2.4 (kbuild
2.4), especially with modules. There are problems with exported
symbols, especially with modversions, there is even a FAQ entry.

I started trying to clean up kbuild 2.4 to remove the module problems
but kbuild 2.4 has reached its end of life, it is complex, buggy,
difficult to maintain and difficult to use correctly for anything but
the simplest cases. All the kbuild 2.4 maintainers agreed that it was
too difficult to continue with, so we started on kbuild 2.5 as a
complete redesign. We did the wishlist and requirements documentation
first [2], before we did any coding - heresy!

[2] See "History" on the Sourceforge Project File List page linked above

Eric Raymond, assisted by the existing CML1 maintainers, did a new
configuration mini language, CML2.
The other half of kbuild is the makefiles, generating the commands for
make, tracking changes and deciding what needs to be recompiled. The
kernel has unusual requirements, we want kbuild to detect command
changes, patches, config changes, link order changes and only recompile
and relink what is absolutely necessary. The standard make code cannot
cope with those requirements, so kbuild 2.[45] do most of the work.

Eric and I did a presentation on kbuild 2.5 at the 2.5 Kernel
Developer's Conference, it went down pretty well. Linus has agreed
that the 2.5 kernel will cut over to CML2 and kbuild 2.5 early in the
2.5 kernel cycle.

I joined SGI in early 2000. They had just lost the person maintaining
the SGI Kernel Debugger (kdb) and I mentioned that I was interested in
kernel debugging, next thing I know I was the kdb maintainer. Dega vu,
add kdb support for better debugging of modules, with modutils changes
to support kdb.

I maintain kdb on ix86 and ia64, these are the Linux architectures that
SGI are interested in, other groups have ported kdb to their
architectures. Because kdb is extensible, other kernel groups have
started using kdb to add commands that are specific to their major
subsystems.

In SGI I also work on XFS, mainly in the packaging and debugging area.
The SGI XFS tree includes kdb, it makes our life so much easier, both
in development and in diagnosing customer problems. Right now I am
porting and verifying XFS + kdb to IA64.

Unfortunately Linus refuses to include a debugger in the standard
kernel. Some distributions are including kdb, either on its own or as
part of XFS.

JA:
In September you posted to the lkml, mentioning the addition of
/proc/sys/kernel/tainted into modutils 2.4.9. Since that time, there's been an
ongoing discussion on the lkml about the merits of such a feature. What is the
purpose behind this change?

Keith Owens:
The original idea came from Alan Cox, he added MODULE_LICENSE to help
him triage bug reports instead of relying on users to say what they had
loaded. Since that needed modutils support I did insmod. At the same
time, some developers wanted to restrict exported symbols to GPL only
code, that also required changes to modutils. A full explanation can
be found at

  http://marc.theaimsgroup.com/?l=linux-kernel&m=100337558817268&w=2

Alas my mail did not affect the signal to noise ratio, people still
continued with incorrect statements about the change.

JA:
How will kbuild 2.5 affect me as a Linux user when compiling new kernels? It
sounds like on the user end things will be easier?

Keith Owens:
The configuration mini language (CML) and the actual build are almost
completely independent. The output from CML is the input to the main
build, you can change CML without affecting the main build and vice
versa.

ESR's changes to CML are listed in http://www.tuxedo.org/~esr/kbuild/,
in particular ANNOUNCEMENT. CML2 is much easier to use for beginners
without sacrificing the flexibility needed by experts. It supports
auto configuration of kernels based on hardware probing. It removes
the CML1 bugs that let users generate inconsistent configuration data.

Once the configuration data has been built by any CML, kbuild 2.5
builds a global Makefile from Makefile.in fragments in each directory
then compiles and links only the sections of code that absolutely need
to be recompiled. The output is a lot quieter, with one line for each
compile or link, that makes it much easier for users to spot problems.

So kbuild 2.5 is easier to configure and easier to build. Once I solve
the performance problems in the build stage, it will be faster as well.

JA:
How will it affect those hacking on the kernel and writing modules?

Keith Owens:
For existing code, not a lot. ESR has converted the CML rules, I am
converting the Makefiles. Adding a new object with its own configure
option requires

  Configure.help text
  One line in CML rules.
  One line in a Makefile.in

Much the same as now. It is just the format of CML and Makefile.in
that has changed, both are simpler and more robust.

kbuild 2.5 makes it much easier to debug code changes. The global
Makefile lets kbuild recompile only code that has been affected by the
changes. You can also tell kbuild 2.5 not to compile the entire
kernel, you can tell it to compile everything in a directory, with or
without recursion, you can even compile individual objects for a fast
check on your changes.

JA:
Why isn't XFS part of the main kernel tree? How about the -ac branch?

Keith Owens:
XFS has changes to the main kernel code, to support direct I/O and
delayed allocation. Delayed allocation gives a great speed up on files
that are created, written to, read then erased. If there is enough
memory, delayed allocation never touches the disk, saving I/O is good.

Unfortunately the delayed allocation changes hit the VM subsystem. Due
to the continual changes in VM we have not been able to get our changes
accepted. We are resigned to waiting for 2.5 and maybe backporting XFS
into 2.4 later.

The -ac branch is even worse, with a different VM entirely. XFS does
not support the -ac tree, although KDB does. Having said that, RH
7.[12] kernels have large chunks from the -ac branch and SGI do XFS
patches against RH 7 kernels. We just do not track the -ac tree on a
daily basis for XFS. I track -ac changes for kdb and kbuild 2.5.

Some distributions are already shipping XFS with 2.4 kernels, Mandrake
8.1 does. SGI do an add on installer based on RH that provides XFS and
2.4 on current RH distributions. AFAIK Debian also have XFS packages.

JA:
I currently use ext3 for two main reasons: compatibilty with ext2, and it's
included in the -ac branch. How does XFS compare? How hard is it to switch from ext2 or ext3 to XFS?

Keith Owens:
To convert to XFS you have to backup your data, reformat the partition
and restore. This applies to any file system change except ext2/ext3.

JA:
How hard would it be to revert back?

Keith Owens:
Since you have to backup and restore, you don't want to switch every
other day. Switching back after you use the extra XFS facilities like
ACLs and DMAPI is painful. SGI are working with other groups to
develop a standard set of ACL interfaces and tools which will make it
easier to maintain multiple file systems and move data between them.

JA:
Why does Linus refuse to include kdb?

Keith Owens:
http://www.lib.uaa.alaska.edu/linux-kernel/archive/2000-Week-36/0575.html

JA:
Why should it be included?

Keith Owens:
http://marc.theaimsgroup.com/?l=linux-kernel&m=96865229622167&w=2

JA:
I've used gdb plenty debugging C code. How does kdb differ from gdb?

Keith Owens:
kdb is designed for a much lower level of kernel debugging. When a
kernel drops into kdb, all interrupts are disabled which means no disk
or network I/O. Disabling all interrupts is the only way to ensure
that you can look at a kernel problem without the data changing under
you. The disadvantage is that you cannot read from disk to get the
symbolic information required by gdb.

kdb can run over a serial console or a PC console. The recommended
debugging environment uses two machines over a serial console, one for
debugging the problem, the other for looking at source code. We looked
at running gdb on the second machine taling to kdb, but the gdb serial
link protocol was not flexible enough. The changes to allow better
scripting in gdb 5 look promising, I plan to review the gdb/kdb
interface (one day).

JA:
What main tools do you use when developing?

Keith Owens:
vi, gcc, gdb, PRCS, kdb. I use my laptop (XFS of course) as the
primary source repository using PRCS, with backups on machines in other
locations. I daily praise Andrew Tridgell for writing rsync, it makes
backups so much easier. Also praise to Josh MacDonald for PRCS, my
kernel 2.4 tree has 700+ patch sets in 10 major branches, I need
something better than CVS to track all the changes.

My home has several ix86 machines, single and dual processor. All
running Linux except for the games machines for the kids. At SGI I
have access to an assortment of machines, including MIPS and IA64.
SGI have plenty of grunt for compiling and plenty of test machines to
be bounced at a moments notice. It is a great working environment.

JA:
What other operating systems do you use? What do you like and dislike
about them, compared to Linux?

Keith Owens:
Assorted mainframe systems on ICL (Maximop, George 2, George 3, VME),
IBM (VS1, MVS, OS/390), Fujitsu (MSP). I still dabble in IBM, I
recently did a performance system for IBM's Unix System Service (Unix
under MVS, wierd). I maintain Infozip for MVS when I can get access to
a suitable mainframe.

ICL's George 3 was years ahead of its time. A proper directory based
filesystem, with eveything integrated into the filesystem. A directory
entry contained the name, who could access it and where the backup
copies were. You could even migrate older entries to tape, leaving
just the directory entry, when you accessed the file it would
automatically mount the tape and restore it. You also got the source
to the OS. This was all in 1975!

Linux filesystems are only just getting to this stage with Access
Control Lists in a couple of systems and automatic archive and recovery
in others. AFAIK, XFS is the only one that has both, with ACLs and
DMAPI.

MVS is mind numbing, everything is a special case. It shows its 360
origins where it started as single user, no OS worth talking about. If
you wanted some system data you had to get it yourself by following
system control blocks. As 360 was replaced by 370, 390 etc. and IBM
introduced VS/1 then MVS, IBM kept the "find it yourself" design.

Eventually IBM started to hide the system data and provide standard
interfaces for obtaining data. Alas a lot of those interfaces were
released as Object Code Only (OCO) with minimal documentation so only
IBM could use them fully, not third parties or customers. I firmly
believe that OCO was a large factor in IBM losing a chunk of the server
market, customers migrated to systems that they could control.

JA:
I was only one then, so my memory is limited. ;^) What happened to ICL and
their George operating systems?

Keith Owens:
IBM had better salesmen and lots of FUD. ICL bundled everything into
one package, buy the OS and you got backup, recovery, online access
(remember this was the days of batch systems), databases, the lot. IBM
separated the components which made the initial offer look cheaper.

  We can sell you an IBM machine with twice as much memory for half the
  price!  Oh, you want backup and recovery, that costs extra.  You want
  online access, that costs extra.  You want a database, that costs
  extra.  You want compilers, that costs extra!

One company converted from ICL 1900 with George 3 to IBM 370 with VS/1.
I did the systems programming and database administration for the ICL
on my own, if I spent more than two days a week on the OS then
something was badly wrong. With the 370 we ended up with four system
programmers and two database administrators, going flat out.

The George operating systems were for the 1900 series. Later ICL
brought out the 2900 series and a new OS called VME/B. The 2900 had a
lovely hardware design, multiple rings of protection at both user and
OS level. Ring 0 was the dump and debug code so even if higher levels
of the OS died you could still find out what was wrong. Arrays were
handled by hardware descriptors which included a length field. If you
tried to overrun a buffer then the hardware would stop you.

Alas ICL lost the FUD war with IBM. Last time I checked, ICL was
basically an IT service company.

JA:
What reflections can you offer on the current state of the 2.4 kernel
series? Do you consider it to be stable?

Keith Owens:
The core of the kernel is stable, apart from bobbles like the recent
(2.4.10) VM changes. Some of the the individual drivers are not being maintained
but that is almost guaranteed with a distributed maintainance system.

My firewalls are still running 2.2 with all current security patches.
I won't upgrade those to 2.4 until ifpw/ipchains/iptables/"whatever
Rusty calls it this week" has had another couple of months testing.

JA:
Why not run OpenBSD for your firewalls?

Keith Owens:
Because I don't have time to learn yet another OS. I am quite happy
with Linux security, I'm just not upgrading my critical systems until
I have to.

JA:
What do you most look forward to in the upcoming 2.5 kernel? When do
you expect work to start here?

Keith Owens:
kbuild 2.5! The sooner that gets in the better. It will cause a lot
of problems initially, especially on architectures that have not been
fully tested. Once the conversion problems are out of the way, kbuild
2.5 will remove a lot of the kernel build problems.

kbuild 2.5 will also make it possible to do things that we cannot do
now. We have two ways of passing parameters to kernel code, the boot
command line (for built in) or modules.conf (for modules). The two
methods are not compatible, each developer needs two chunks of code,
selecting one depending on how the code is compiled. kbuild 2.5
provides the missing piece and will give a consistent way of specifying
parameters.

Once kbuild 2.5 is in, I will be starting on modutils 2.5. That will
get rid of a lot of the compatibility crud, modutils currently supports
kernels back to 2.0 which causes problems when I want to change
anything. There are modutils bugs that I cannot fix without affecting
existing users, the overriding constraint on modutils is that I cannot
break backwards compatibilty, I don't want to be lynched. Modutils 2.5
will be a complete redesign and will break backwards compatibility, so
it can only be used on a 2.5 kernel. Obviously it will fall back to
modutils 2.4 if you boot a 2.4 kernel.

There is also 2.5 work planned to remove the module unload race without
add MOD_INC_USE_COUNT everywhere. Rusty Russell and I have been
discussing this but, for me, it comes after kbuild 2.5 and modutils
2.5.

I wish that some of the architectures were more integrated into Linus's
tree. But without a proper development kernel I can understand why
each arch group is keeping separate trees. We need a development
kernel to test the integration before backporting to the stable line.

When we will get a development kernel? No idea.

JA:
Is there much resistance to rewriting modutils in a way that's not backwards
compatible?

Keith Owens:
Since it will fall back to modutils 2.4 on older kernels, nobody can
really complain about backwards compatibility. I am sure that I will
get some complaints about the new file formats but that is what
development releases are for.

JA:
Have you met Linus? Alan? Other kernel hackers?

Keith Owens:
I met Linus years ago in Singapore, and have met various hackers at the
Australian Linux conferences. The 2.5. Kernel Developer's conference
in San Jose was a great opportunity to meet most of the kernel
developers and put faces to names.

http://marc.merlins.org/linux/linux.conf.au_2001

http://lwn.net/2001/features/KernelSummit/

JA:
What was your impression of Linus?

Keith Owens:
Despite his claims to be a conniving bastard, I found Linus to be
quite the contrary. He does not suffer fools gladly and will pick up
on the weak points you hoped nobody would notice, but give him a good
technical argument and he will listen.

JA:
Can you offer more reflection on the 2.5 Kernel Developer's conference?

Keith Owens:
Although the talks were useful, the biggest advantage was getting
together around a whiteboard and arguing technical points. Email is
fine up to a point but sooner or later you need a whiteboard. I find
that local conferences are useful for the same reason, get a few kernel
hackers together and you can sort out a lot of problems very quickly.

JA:
What do you enjoy doing in your non-Linux time?

Keith Owens:
Skiing. Two years running at Whistler/Blackcomb was great, I highly
recommend Mike Dempsey's course. Austria was a nice break, just before
they had the avalanches. The Australian mountains are fun when we get
a decent dump, but I got spoilt by Whistler.

Ten pin bowling, average 165.

Scuba diving. I persuaded my daughter to learn, now for my son.

Science fiction. My library is around 3,000 books, one day I will
catalog it.

JA:
How advanced a skier are you?

Keith Owens:
Black to double black. I don't do off lift runs, I am too lazy to
climb up hills myself. Apart from that I will try almost anything.

JA:
Have you ever dived off of Belize? It has some beautiful reefs. But then,
Australia is supposed to have everything beat - I've not been there yet.

Keith Owens:
No, Malaysia, Thailand, UK (murky), Australia. The best was on Heron
Island, directly on the Great Barrier Reef. With most diving you spend
a couple of hours on the boat each way before you can dive. At Heron
Island you just sail out of the harbour, drop anchor and go over the
side.

JA:
I also quite enjoy science fiction. What are some of your favorites?

Keith Owens:
I change from one author to another over time. Right now it is Vernor Vinge, Lois McMaster-Bujold, David Weber, Larry Niven, Roger Zelazny,
Terry Pratchett and Iain M. Banks (but not Ian Banks). Mainly hard
science fiction, a man can only take so many elves, dragons and rings
of power, most fantasy seems to be a pale imitation of LOTR.

Keith Owens:
I change from one author to another over time. Right now it is Vernor Vinge, Lois McMaster-Bujold, David Weber, Larry Niven, Roger Zelazny,
Terry Pratchett and Iain M. Banks (but not Ian Banks). Mainly hard
science fiction, a man can only take so many elves, dragons and rings
of power, most fantasy seems to be a pale imitation of LOTR.

-->

JA:
What tips and inspiration can you offer aspiring kernel hackers?

Keith Owens:
If you have an itch, scratch it. It does not have to be a big problem
or patch, pick something small and fix it. I started on ksymoops and
grew from there. If you cannot contribute code, document. Anybody
fancy rewriting the Modules HOWTO for me?

JA:
Where does the most up-to-date Modules HOWTO reside?

Keith Owens:
The one on the net is well out of date. The previous maintainer sent
me the raw files and all the updates he had but both he and I ran out
of time. I have the collection for somebody to clean up and correct.

JA:
Is there anything else you'd like to add?

Keith Owens:
Fight spam, software patents and the DMCA/SSSCA. I'm with Alan on this
one, I will not be attending any more conferences in USA while the DMCA
is in force.

JA:
Thank you very much for all your thorough answers! They have been an education
for me.

---

About the interviewer: Jeremy Andrews was born and raised in Southeast Alaska. Currently he lives and works in South Florida. He maintains KernelTrap as a hobby.