This week KernelTrap interviewed Robert Love who currently maintains the preemptible kernel patch, among other things. He's been using Linux now for about 7 years, with numerous contributions in the current kernel. All of this is best described in his own words...

JA:
Please share a little about yourself and your background...

Robert Love:
I am currently a Computer Science and Mathematics student at the University of Florida in Gainesville, although I am originally from outside Ft. Lauderdale, FL.

I am not married, but I have a wonderful girlfriend.

My programming interests are in operating systems and mathematical/scientific computing.

JA:
When do you expect to graduate? Do you have post-graduation plans?

Robert Love:
Expected graduation date is 2004.

I want to stay in school for a graduate degree, especially if the economy remains as-is.

JA:
Where outside of Ft. Lauderdale? That's where I live now, in Lauderhill.

Robert Love:
Born and raised in Pembroke Pines.

JA:
How and when did you get started with Linux?

Robert Love:
My first personal computer running Linux was in late 1994, on a 1.0 kernel. This was my first real experience with a UNIX, too. I had a 386SX (thanks Mom!) and switched from a beta of Windows 95. I used Linux progressively more through 2.0, and finally switched completely during 2.2. I've been all Linux ever since.

I made the switch partly for the obvious reasons: I believe Linux is a powerful and robust system with good applications. The greater reason, however, was I love the ability to play with the system and the community that surrounds that.

JA:
Can you describe a few key differences between the 1.0 kernel compared to the current 2.4 kernel?

Robert Love:
I would be surprised if 1.0 and 2.4 shared more than a couple pages of code. So, everything is different.

JA:
What was it like to install and use the 1.0 kernel? What
software was even available?

Robert Love:
Thankfully, come 1994/1995, Linux was evolving nicely. At this time SLS
and Slackware were both making solid full-featured distributions. I
first used Slackware 2.0. You had to download the thing in disk sets --
maybe you still do. I remember disk set A was like the core, and disk
set N provided networking, etc.

By this point Linux had networking, X worked (XFree86 2.0!), and a
decent amount of hardware was supported.

The biggest problem from today was the lack of documentation, which was
compounded by the fact things were fairly complicated to get setup. I
remember I couldn't get PPP working... I eventually gave up. If I could
of gotten PPP working, I may of switched to Linux then!

JA:
What are some of the contributions you've made?

Robert Love:
Right now most of my time is going into maintaining the preemptible kernel patch. This patch allows a lower priority process to be preempted, even if operating in kernel space, resulting in improved system response. The patch was originally the work of MontaVista - a great company - so I am working closely with them and the rest of the community. It is a fun project. We are aiming for inclusion in 2.5.

I'm pretty random, otherwise.. I fix bugs, optimize, and clean cruft up. I wrote the i815 and AMD761 AGP GART code. I also maintain some other odds and end patches.

JA:
What kinds of changes have to be made to the kernel by your patch to allow for kernel preemption?

Robert Love:
The model we use is to allow the kernel to be preempted at any time when
it is not locked. Under this design, when an event occurs that causes a
higher priority task to become runnable, the system will preempt the
current task and run the higher priority task.

We had to modify the interrupt code in entry.S to prevent some
situations and to allow preemption on return from an interrupt handler.
However, we can't preempt within critical regions for the same reason we
can't allow concurrency within them with SMP -- so we prevent preemption
while holding a spinlock. The bottom half handler and scheduler were
also modified to prevent preemption while they are executing.

JA:
How stable do you consider your patch, as it stands today? I do all of my
work on a single Linux server, which I am continually upgrading with new
kernels and new software. Improved overall system response is certainly
attractive, but do you consider the patch stable enough for an
'every-day-use' server?

Robert Love:
The patch is very stable. I would feel confident suggesting anyone use
it. We have a lot of users and are receiving a lot of feedback. I
think we have the obvious bugs taken care of. I use the patch on my
main system.

The good thing is we know the design and implementation is right. With
our recent patches, I feel we have taken care of any problems.

JA:
How significant an improvement in system response time do you see?

Robert Love:
Very significant.

A 200% improvement in system latency has been noted. With the
preempt-kernel patch, average system latency is now around 1ms, with
spikes up to 10ms. There are still a few long-held locks that kill
response time, and some of them are 100ms+. Thankfully they occur in
specific places, like switching VCs.

JA:
There is another patch available (maintained by Andrew Morton) with the same
end goal. How is your patch different?

Robert Love:
The source of the problem both patches are aiming to solve is that
kernel code is non-preemptible. Thus, it runs until it completes or
explicitly yields control. If userspace events happen during kernel
operations, they must wait until the kernel finishes. This waiting is
what causes the system latency we are aiming to fix.

Andrew's low-latency patch places conditional scheduling calls in
strategic locations throughout the kernel. These calls effectively
allow long kernel operations to say "I will be nice and let someone else
run. Any task need to execute? If so, run!" This breaks up the
operations, reducing latency in those places.

The preempt-kernel patch works much differently. It modifies the kernel
itself to allow kernel code to be preempted. So now the above problem
doesn't even exist - if a task needs to run, it will run - and system
response benefits.

JA:
In a recent lkml thread you talked about possibly combining the two patches.
Is this something you're working towards?

Robert Love:
It is certainly something to consider. Since we can't preempt during
locks, the longer held locks pose a noticeable threat to system
latency. The duration of the lock effectively becomes the latency of
the system for that period.

One solution is to break the locks like in Andrew's patch. This would
be the purpose of merging parts of his patch with preempt-kernel, or
more realistically making a preempt-aware version of his low-latency
patch. Any of his conditional schedules that occur while a lock is held
may be useful.

One step in this direction is identifying the long held locks. This is
what the preempt-stats patch does. It measures periods of disabled
preemption and reports what lock caused it.

This is just one solution to the issue of long-held locks.

JA:
What are some sections of the kernel you've identified to have these long-held
locks?

Robert Love:
The console layer used to hold some locks for a terribly long time, but
thankfully Andrew Morton fixed a lot of that recently. The frame buffer
code disables interrupts for a long time -- avoid printing a lot to it.
If you scroll a lot, you will have latencies even up to 500ms.
Switching virtual terminals is bad. blkdev_close() holds a lock a long
time. Some module operations take awhile. If the VM starts thrashing,
it causes a lot of latency, since it does all this thrashing while
holding a lock. VFS has a few long-held locks, too.

JA:
How realistic is it that your patch will get into 2.5? What are some of the
obstacles?

Robert Love:
It all depends on Linus's opinion of the patch, and what the other
hackers argue for. Linus before has suggested he is interested in the
idea, I hope that interest helps get the patch merged.

We have a very large userbase for an independent patch and that
obviously helps a lot. We have gotten a lot of feedback including
benchmarks that show strong improvements.

The counterargument is that a preemptible kernel lowers system
throughput. This is a very valid point, and one we will need to
address. Tests now show a loss in throughput of 0-5% -- I think that is
worth 200% increase in response! More so, some scenarios show an
increase in throughput, since we thread better. Despite this, for
systems where any loss of throughput is unacceptable, the preemptible
kernel is a config setting.

JA:
What main tools do you use when developing? Please describe your environment(s), computer(s) and the methods you use.

Robert Love:
My main machine is a P3-733 with 384MB RAM and U2W SCSI disks. I also have an IBM ThinkPad and an assortment of older stuff lying around the apartment. I run the latest RedHat Rawhide with Ximian GNOME.

I code in vi because I don't want to learn another OS. :)

I am typically in X, so I usually have a handful of gnome-terminal and gvim windows open with my current project. I tend to grep around, research as needed, and then start coding. I rarely do design on paper. I rely on lkml and personal email often -- I like bouncing ideas off others.

JA:
What other operating systems do you use? What do you like and dislike about them, compared to Linux?

Robert Love:
I have a workstation with Windows 2000 and an old SGI Indy running Irix. Otherwise everything else is Linux. I keep Windows around for Powerpoint and in case AbiWord just doesn't cut it.

The greatest dislike about the other systems is that they are not open source. I am not a free software zealot - although I praise its merits - but the open availability of source and the community surrounding that is priceless.

JA:
Have you played with any of the open source BSD operating systems?
Specifically, have you looked much into their kernels?

Robert Love:
I used to have an OpenBSD machine but I thew it into a dumpster (not
because of OpenBSD, though).

The rest of my knowledge is academic. I am impressed by Matt Dillon's
work in FreeBSD (he wrote the VM among other stuff). Their SMPng work
is coming along nicely (they lag Linux in SMP scalability) and they are
even considering implementing a preemptible kernel.

JA:
Do you know what form of preemptibility they intend to use? Will it be like
used in your patch, or Andrew's patch, or perhaps a combination?

Robert Love:
They are considering a fully-preemptible kernel, like what my patch
does. I don't know what they have done towards this, other than it has
been discussed by the FreeBSD Core team.

JA:
What reflections can you offer on the current state of the 2.4 kernel series? Do you consider it to be stable?

Robert Love:
I think we have come an amazing distance since 2.2. The work done during 2.3 and now in 2.4 is incredible. Some of the advancements that we did, especially some of the work by Linus and Ingo, are incredible. 2.4 can play in some very high-performance/high-scaling playgrounds now.

This is not to say I am completely happy with 2.4. At this point, I think ripping out Rik's VM was a mistake. If Linus wants to try something different in 2.5, by all means he should. In the stable series, however, I really think we need to make much smaller measured changes. We didn't just throw away a year plus of VM work, but we also tossed out all the corresponding documentation and understanding. Nonetheless, it seems the VM is shaping up OK. I use Alan's kernel, which has stuck to Rik's VM and we are working on improving that. Who knows what VM we will have when 2.4 is handed to Alan.

JA:
If you were suddenly in charge of the kernel, how would you handle these
issues?

Robert Love:
First, the kernel would degenerate into a pile of cat phlegm with me at
the helm. For whatever complaints I or others have against Linus, he is
not just an incredible hacker but a superb manager. Linux needs him.

On the issue of the VM, I would follow Alan's lead for 2.4. Rik's VM is
tested and documented. I think we needed better testing of it. Too
many of Rik's ideas and patches were ignored. Other patches were merged
that shouldn't of been. We needed to merge VM work little-by-little,
alongside no conflicting patches, and test the results. We have
something close to this in Alan's tree now.

For 2.5, of course, anything goes. Rip out whatever. I think Andrea's
VM is performing well. Maybe it is better. It is just a bad time to
find out, IMHO.

JA:
What do you most look forward to in the upcoming 2.5 kernel? When do you expect work to start here?

Robert Love:
Of course, a preemptible kernel :)

Beyond that, I'd like to continue the quest for finer-grained locking and overall cleanup. I think a block device, SCSI layer, and console layer rewrite is planned. We should see NAPI and/or some form of IRQ throttling. Ben LaHaise's async I/O work should be merged, too. Keith's new kbuild is looking great. I look forward to all this.

I really think we need a total rewrite of the tty system, but I don't know who is up to that...I surely am not.

I don't think anyone, Linus included, has a clue when 2.5 will start. I originally figured somewhere in March-May -- a couple months after 2.4.0, like usual. Now that a year is approaching (2.4 was released 4 Jan 2001), I can't even guess. I suspect when both the VM proves solid and Alan and Linus are fully synced, we will see a 2.5.0. I hope that is soon.

JA:
Have you met Linus? Alan? Other kernel hackers?

Robert Love:
I haven't met any of the "famous" hackers. I look forward to meeting them, someday. Perhaps at a conference or petting zoo.

JA:
What do you enjoy doing in your non-Linux time?

Robert Love:
Outside of Linux, school, and the girlfriend I like mountain biking, tinkering with my car, and going to rock shows.

JA:
What are a few good shows you've seen recently? Gainsville certainly has had
a lot of famous music come out of it.

Robert Love:
Hm, not much recently. Cadillac Blindside did a show a couple weeks
ago, I like them. Jane's Addiction played last night. Of course, Less
than Jake is local, so I see them a lot. They are great.

JA:
What tips and inspiration can you offer aspiring kernel hackers?

Robert Love:
Read the source, play with the source, and bathe regularly. For beginners, O'Reilly makes some superb books ("Understanding the Linux Kernel" by Bovet and Cesati and "Linux Device Drivers, 2ed" by Rubini and Corbet are excellent). But the key is to read the source. You have the source to the whole system -- read it, change it, learn it.

I subscribed to lkml years ago, but I didn't make my first serious contribution until 2.3. Lurking on lkml is a great way to see The Greatest shoot it out and you can learn a lot from that.

JA:
Thanks for taking the time to thoroughly answer all my questions! Because of
our conversation, I am definitely going to apply your preemptible kernel patch
on my system.

Robert Love:
Great! Let me know how it goes and thank the great guys at MontaVista,
too.

Follow up:
After talking with Robert, I dowloaded his patch and gave it a test run (applied to 2.4.12-ac1). Wow! The improvement is amazing! Programs like VMWare and Mozilla used to freeze up my system for a short while during launch and exit. With Robert's patch, I can launch them both at the same time and my system remains completely responsive. Also noticed that xmms no longer "skips" while these big programs launch. There's no question that I'll be using this patch always, now. I hope it gets merged with the main source in 2.5.

Any comments you have about this interview, or this series, can be posted here or mailted to me at jeremy@kerneltrap.com