This week, KernelTrap has spoken with Linux kernel hacker Rik van Riel. Rik is perhaps most recognized for his impressive rmap VM efforts. He's also the founder of kernelnewbies. Living in Brazil, he works for Conectiva.

---

Jeremy Andrews: Please share a little about yourself and your background...

Rik van Riel: I was born in 1978, in the north-eastern part of Netherlands, where I grew up on my father's farm and spent most of my life. As a teenager I used to do farm work in the holidays, which allowed me to buy myself a computer. I started with DOS, played with OS/2 for a while when I got my 486 in 1994 and went to Linux later that year.

In the beginning of 2000 I accepted a job at Conectiva, the largest Linux company in South America, and moved to the city of Curitiba in the south of Brazil. I'm working full-time on Linux now, mostly development but also helping out Conectiva's customers when problems happen.

JA: Can you describe a typical work day at Conectiva?

Rik van Riel: This really depends. Lately my typical work days are at home, where I wake up between 7 and 8 in the morning and start working on the kernel. I help the consultancy and support folks over irc and email and do what every full-time free software developer does.

Then there are the un-typical work days, when something breaks on a server of an important client and the consultants can't figure it out. In that case Conectiva will ship me over to the client's site and I won't return home until the problem is fixed. Luckily Conectiva's consultancy team is pretty competent so these emergencies almost never happen.

JA: How did you get started with Linux?

Rik van Riel:In early 1994, some friends from school convinced me to try Linux on my 486 once it turned out that 4MB wasn't quite enough memory to run OS/2 comfortably. With their help I installed Slackware 2.something, which came with kernel 1.1.59. I remember the first weekend, with just the A and AP floppies installed ... a Linux system without many applications, so I spent my time reading man pages and stressing the system.

After picking up shell scripting and some simple programming it was time to move on to larger stuff. I ended up looking at the kernel source for two simple reasons, firstly because I was curious as to how Linux worked and secondly because that was the only source code I had installed (I had no internet connection, not much disk space and kernel modules didn't exist yet, so it was necessary to compile your own kernel and have the kernel source).

JA: What are some of the contributions you've made?

Rik van Riel: Apart from the obvious random bugfixes my main contributions have been to the memory management and the scheduler. I have also made a number of "social" contributions, like NL.linux.org, linux-mm.org and, most importantly for the KernelTrap audience, the kernelnewbies project.

JA: When did you start kernelnewbies?

Rik van Riel: I think I started kernelnewbies early in 2000, just before I moved to Brazil. The channel was started after I got the same questions about the kernel on irc every day and I found myself answering the same questions over and over again to different people. The first half year or so I was still the only one answering questions to the 10 people resident on the channel, but after that things started growing.

JA: How involved are you in the project these days? How many others are involved?

Rik van Riel: Right now I'm in the #kernelnewbies irc channel pretty much all day and I still answer lots of questions by people. Luckily there are some more people online to help answer questions though, the #kernelnewbies irc channel usually has between 200 and 300 people online and the kernelnewbies mailing list has some 600 subscribers.

JA: Can you talk a little about what is offered at kernelnewbies?

Rik van Riel: The kernelnewbies project is meant to help people learn about the kernel. Usually this means discussion about the Linux kernel, but of course people wanting to discuss other operating systems are welcome too.

Kernelnewbies currently has a web site, an irc channel, a mailing list and a wiki. More importantly, it has an active community of people who are willing to learn and help each other learn.

JA: What insight can you offer regarding Linus' move away from your VM to Andrea's VM, well into 2.4 development?

Rik van Riel: Linus is very good at changing his mind ;)

JA: How receptive is Linus to putting your VM back into the kernel, during 2.5 development? Is this a goal for you?

Rik van Riel: Linus has expressed interest in merging my VM work into 2.5 in small pieces. I've already started extracting the first part of the VM work and submitting it to Linus.

JA: You continue to develop your VM, frequent updates being made available. How has the VM changed since it was part of the main kernel? How stable is it?

Rik van Riel:The fact that my VM code is no longer in the kernel has two positive effects. First I don't have to worry too much about stability or code freezes, I can just develop the code in the direction of better quality without having to be afraid of changes. Secondly I can integrate bugfixes much quicker.

This has lead to my reverse mapping based VM becoming stable within a few months of me picking it up again; it is now stable to the point where it survives the kernel torture lab Bob Matthews is running at RedHat and both Alan Cox and Michael Cohen have integrated the patch into their kernel tree. I suspect the rmap VM isn't as fast as some of the other VM code out there, but it does seem to hold up somewhat better in strange situations.

JA: What is your intention for this VM system?

Rik van Riel: Making it the best VM subsystem I can. I have some items on the TODO list, but chances are that other people will come up with patches and/or requirements I haven't thought of right now, so things are always open to change. Now that the VM is no longer in the kernel I can also clean up the code, in particular getting rid of some of the locking horror we have built up over 5 years of SMP development. In places the SMP locking is so subtle that you cannot change any small part of the code without first needing to understand all of the code, this really needs to be fixed so the code is a little bit more modular.

JA: Is the modularity something that you are now working on?

Rik van Riel: More or less, one first part of modularity is cleaning up the code and letting eg. device drivers only access the VM through interfaces and not letting them mess with internal data structures of the VM.

Of course, this is all source modularity we're talking about, so in many places it is good enough to abstract something away into a simple define in a header file. Modularity in the binary object really isn't required for anything.

JA: What are some of the major differences between Andrea's and your VM?

Rik van Riel: Andrea's VM is building on the VM work done during the 2.3 kernel and seems focused on tuning and tweaking that VM till the limit. I suspect he is getting close to the maximum performance that can be achieved with that VM design.

My VM hasn't undergone much performance tuning yet. I've looked very carefully at the limitations of the design of the VM in 2.3 and am trying to attack these fundamental problems.

One example would be memory zones. For machines with multiple memory zones you really want the VM to be able to free memory from one particular zone. Another issue would be balancing the eviction of cache vs. the eviction of memory used by processes. Both of these problems arise from the fact that traditionally the Linux VM was scanning page tables of processes in the pageout code.

The reverse mapping VM fixes both of these issues (and a few more) because it knows all the users of each physical page, so it can evict pages from the right memory zone and balance the eviction of process and cache pages.

JA: Do you continue to provide documentation for your VM, as you did when it was included in the 2.4 tree?

Rik van Riel: Absolutely. If I do not document what the code is supposed to do, it will be much harder to identify and fix bugs. From code without documentation you can only know what it does, but you have no idea if it's really supposed to be doing that.

Documentation also helps getting people interested in how the code works. This leads to beta testers and sometimes even developers helping with the implementation of the rest of the code (make sure to document some things you haven't implemented yet).

JA: Is your rmap patch for 2.4 compatible with Ingo's scheduler patch?

Rik van Riel: Yes.

JA: How about with Robert Love's preemptible kernel patch?

Rik van Riel: Yes.

JA: What is your current involvement with with the NL.linux.org spamfilter? What exactly is the spamfilter and how is it used?

Rik van Riel: I started NL.linux.org's spamfilter project by putting the anti-spam regexps used for the mailing list in CVS and making a quick script to generate the majordomo.conf. Other people then got access to the CVS tree and started writing scripts to also generate spam filters for procmailrc, mailagent and smartlist. Once NL.linux.org's mailing list software was converted to Listar it was easy to just have the spam-regexp file generated from the same set of regexps.

Basically the project at http://spamfilter.nl.linux.org/ is a set of regular expressions used to block spam. The regular expressions are added manually, but that's OK since about a dozen people have CVS commit access. It is a way of dividing the work of blocking spam.

JA: Who uses this service?

Rik van Riel: There are maybe 50 to 100 users who use spamfilter from their procmail rules, the mailing lists on NL.linux.org and a few other servers are protected by spamfilter and I know of at least one ISP using the filter.

JA: What main software tools do you use when developing? What hardware do you use?

Rik van Riel: I use vi and bitkeeper, sometimes usermode linux with gdb. My testing machines are a dual Pentium at 120 MHz with 64 MB of RAM and a K6-2 500 with 512 MB of RAM. The slow CPU speeds make it easy to identify any places in the VM where we are using too much CPU. My desktop machines are a Celeron 433 at work and a K6-2 500 at home.

At the office there also are a few big machines with high memory, but those are used for all sorts of things so I don't use them for testing too often.

JA: How long have you been using BitKeeper? What is your impression of the tool?

Rik van Riel: I've been using BitKeeper for some 4 months now. I like it a lot because this is the first source control system that actually reduces the amount of work I have to do when porting a patch from one version of the kernel to a newer version.

I've tried CVS in the past, but it only made life harder, not easier. I hope BitKeeper will also provide some inspiration to the free software fanatics, some encouragement to implement a free tool with some of the same functionality. BitKeeper should be a nice challenge for the source control people...

JA: What operating systems other than Linux have you used? Do you still use them? How do they compare to Linux?

Rik van Riel: I've tried various flavours of DOS, OS/2, Linux, FreeBSD and VSTa. However, I haven't really used any other OS than Linux since 1995, so I can't speak from recent experience.

Modern FreeBSD should be a bit better than Linux under heavy overload situations though, while Linux should have better SMP scalability. The good news is that people from both camps are working on these issues.

JA: Why do you say that FreeBSD's VM should perform better under heavy overload situations?

Rik van Riel: FreeBSD has some special code to reduce the load on the VM when the system gets under high load. They limit the rate at which pages become eligible for swapout and start suspending processes when the combined working set of all running processes won't comfortably fit in memory. Linux still lacks these features.

JA: Did you know Marcelo Tosatti prior to his taking over 2.4 kernel maintenance?

Rik van Riel: Marcelo is one of the people who convinced me to work for Conectiva. Once I moved to Brazil I even stayed at his house for a few months while searching for my own place to live. When we happen to be at the office at the same time (we both work very liberal hours) we sit next to each other. I guess you could say I know Marcelo ;)

JA: How stable do you feel the 2.4 kernel has become?

Rik van Riel: The 2.4 kernel is getting pretty stable, but like any sufficiently complex piece of software it still has various bugs. Most notably (to me, at least) the VM shows bad behavior in some corner case situations. Some of these are due to bad tuning, Andrea has fixed many of those in his -aa tree. However, some of the other problems are fundamental issues that cannot be solved by tuning the current code, I'm trying to attack those in my -rmap tree.

JA: What other areas in 2.4 do you feel still need effort to get the overall kernel stable?

Rik van Riel: I think most other parts of 2.4 are stable by now. Of course there will always be some drivers with problems, but there's nothing you can do about that when you have hundreds of device drivers in the tree.

JA: Are you more focused on 2.4 stabilization, or 2.5 development?

Rik van Riel: I'm focusing on 2.5 development at the moment. This is mostly because the last problems remaining in 2.4 seem to be fundamental problems which cannot be fixed by just tuning the code. Having said that, I use 2.4 as my development platform because that is a much more stable basis to be testing on. The code which is ready gets ported and submitted to 2.5 in small, controllable chunks.

JA: Do you have any suggestions to people just learning the linux kernel?

Rik van Riel: Look around for interesting kernel patches and apply those to your tree. You'll begin with one kernel patch, which is easy to apply. However, after a while you'll run into other fun kernel patches which you'll also want to use. Often this will just work, but sometimes you need to fix up conflicts between these patches. That is an excellent motivation to start learning about that part of the kernel and start kernel hacking ;)

When you've reached this stage, don't forget to take a look at http://kernelnewbies.org/ and talk to the people in #kernelnewbies ;)

JA: What do you enjoy doing when you're not kernel hacking?

Rik van Riel: I like listening to shortwave radio and often hang out in #swl on the Starchat IRC network. I also enjoy cooking, eating and generally enjoying life with my fiancee, Fabiana.

JA: When is the wedding?

Rik van Riel: We're still busy planning our marriage, but we want to marry at the beginning of the next summer, in November or December. Planning this far ahead is necessary because we'll be inviting friends from all over the world ;)

JA: What do you listen to on shortwave radio?

Rik van Riel: The primary use is listening to Dutch language programming from Radio Nederland Wereldomroep and Radio Vlaanderen Internationaal, but I'm also listening a lot to the BBC, Radio France, and dozens of other broadcasters from all over the world.

I also enjoy listening to radio stations from other regions of the world, for example local stations from the Andes and Africa and some regional stations from Asia and Europe. At times I also listen to amateur radio operators and utility stations.

JA: Are you a good cook? What types of foods?

Rik van Riel: Most of the world's food pales in comparison with Brazilian food, so my skills of Italian, semi-Indonesian and Dutch food aren't too spectacular here. I'm having fun making side dishes for Fabiana's Brazilian food though and am slowly catching up with what has to be one of the most elaborate food cultures in the world.

Just to put things in perspective, I used to think that food from southern Germany and Spain was elaborate and would make me fat. Nowadays I loose weight when I visit Germany or Spain.

Brazil certainly isn't what people would expect from a third world country. Sure there is poverty and large social problems, but most of the normal working people here (at least, in this region of the country) still seem to have a better quality of life than people in western Europe or the USA, with much better food, more living space, more free time and less stress.

JA: Is there anything else you'd like to add?

Rik van Riel:Corporate "culture" is a big threat to science and general progress of human culture at the moment. Traditionally people have built themselves a culture by learning from others and copying things from other people.

In order to encourage people to contribute their knowledge and inventions to the public domain copyright and patents were invented, where publishers and inventors got a temporary monopoly (with legal protection) on their work in exchange for contributing their work to the public domain, where everybody can learn from the inventions and further the knowledge of the human race.

Nowadays companies are trying to undermine this system by trying to make their copyright valid forever and trying to use patents to limit the technical means other people can use (instead of just collecting royalties).

The investors and the management of these companies need to realize that the last 150 years of explosive growth in technology has only been possible because of people learning from each other's discoveries and making use of other people's inventions. I am afraid that if we do not return to the idea of "standing on the shoulders of giants" (as Newton put it so nicely) technological improvements may be slowed to a pace below population growth and billions of people will suffer
the consequences.

It is of critical importance that we adjust the copyright and patent laws to make way for a FASTER growth in technology.

"Personally I wouldn't mind stricter patent regulations for a shorter
period of time or compulsory licensing of technology for a fee determined
by the state. It is just as important to share information as it is to
fund research so in my opinion the law should promote both information
sharing and research funding. Tricks to delay keeping things out of the
public domain should be forbidden."

JA: It was a pleasure talking with you, and I look forward to seeing your rmap VM merged into the 2.5 kernel. Thanks!

---

Related Links:

• Rik's Home Page - (http://www.surriel.com/)
• Conectiva's Home Page - (http://distro.conectiva.com/)
• KernelNewbies - (http://kernelnewbies.org/)
• Rik's Linux Kernel Patches - (http://www.surriel.com/patches/)
• Spamfilter Project - (http://spamfilter.nl.linux.org/)

---

Join our announce-only mailing list to learn when future interviews are posted. To subscribe, enter your email address and a non-valuable password for account management. You will receive a brief email message each time an original interview is posted to KernelTrap. (No more than one message a week)

Your email address:

Pick a password:

Confirm your password:

---

About the interviewer:

Jeremy Andrews was born and raised in Southeast Alaska. Currently he lives and works in South Florida. He maintains KernelTrap as a hobby.