Marcelo Tosatti became the maintainer of the 2.4 stable kernel when he was 18 years old in November of 2001. His first kernel release was 2.4.16 on November 26'th which very quickly followed the earlier 2.4.15 to address an issue with filesystem corruption. Two years later, he has recently released 2.4.23 and plans to soon put the 2.4 stable kernel into maintenance mode, only addressing bugs and security issues.
Living in Brazil, Marcelo currently works for Cyclades Corporation. In this interview he looks at how he became the 2.4 maintainer, the challenges involved, and brings us up to date with the current status of the 2.4 kernel.
[1]Jeremy Andrews: Please share a little about yourself, your background, and how you first started working with Linux...
Marcelo Tosatti: My interest for computers started with influence from my older brother Nei Tosatti (he used to buy spare sparts and and "assemble" computers to sell them... that was around 1994, I was 12 years old). Before that happened, I was an average kid who didn't enjoy school much (I wish I did).
At the time I learned how to use DOS and started playing with BASIC/games/etc. A bit later I got interested in C and learned a bit of it (reading books) and practicing (I remember compiling with "BCC").
I heard about Linux when I first had access to the Internet (around 1995/1996), and I bought "Linux FT" from some company in my hometown. At the time I was working on a local ISP, and I replaced some of the NT servers they used with Linux. Then I had the chance to work with development at Conectiva (where I worked for the next 6 years and got interested in kernel development).
I'm 20 years old now. I lived my whole life in Curitiba, Brasil with my parents, and finished high school there.
In July 2003 I met Suzana and moved to live with her in Porto Alegre (more to the south). At the same time I quit my job and go to Cyclades.
JA: This means you were 13 or 14 when you started working for Conectiva? How did you come to start working there at such a young age?
Marcelo Tosatti: Yes... I was working on the ISP I mentioned when I mailed Conectiva asking for a job and we talked, and I started working with them.
JA: What were some of your contributions to Linux development prior to becoming the 2.4 maintainer?
Marcelo Tosatti: I fixed some VM problems on 2.2 and 2.3/2.4 (most fixes related to the "memory reclaiming logic").
I used to maintain Conectiva's kernel RPM.
JA: You recently started working for a new company. Can you offer any reflections on what lead to this decision, and on how it's going?
Marcelo Tosatti: I left Conectiva because I worked 6 years there, and I felt I needed to change (do something different, work with a different company).
Cyclades offered me a good job position, with nice projects. Cyclades builds special purpose hardware (console servers, power managers, etc) which now run Linux. I'm working with their embedded distro and kernel which are used on these machines (most of them use PPC processors). I am doing some driver work right now.
JA: I remember when Linus was getting ready to hand off the 2.4 kernel, everyone (including Linus) was expecting that he would give it to Alan Cox. How was it that you instead became the 2.4 stable kernel maintainer?
Marcelo Tosatti: Alan decided to stop maintaining and he suggested me. I had the time and the will to do it.
In the beginning it was too much pressure (from the media, etc) but now it got better in that aspect.
JA: Now that you've had quite a bit of experience maintaining a stable Linux kernel, what do you think?
Marcelo Tosatti: Well I think over time I made a lot of progress in my relationships with developers (knowing who you can trust, who you can request for help, etc).
A lot of people seem to dislike my maintainership (www.kerneltrap.org [2] people for example :)), but I think most developers are fine with it.
Not as excellent as Andrew or Alan would do, but still...
JA: There is indeed a small but vocal group on KernelTrap that have said negative things about your maintainership of the 2.4 kernel, however I don't feel that they in any way represent KernelTrap readers as a whole.
Have you received any feedback from Linus, Alan Cox, or Andrew Morton regarding your maintainership of the 2.4 kernel?
Marcelo Tosatti: I asked Linus once about what he thought about my 2.4.x maintainership and he said he didn't have any complaints.
I believe Alan is also OK with it.
JA: What is the biggest challenge for you in maintaining the 2.4 Linux kernel?
Marcelo Tosatti: Well, I believe unmaintained code. All core parts are well maintained, however there are a few drivers and subsystem which are not. I obviously dont know all the code, so...
Also I wish all bug reports were nicely and well reported, but that is not always the case for several reasons (not everyone has the time, etc).
I think there is not much of a big challenge in maintaining 2.4 thanks to all the subsystem maintainers and people from the community who help with 2.4 bugfixing. I could make a giant list.
JA: You've been managing the 2.4 source tree with BitKeeper now for over a year. Has it proved to be an improvement over what you were doing before?
Marcelo Tosatti: Yes. Everything is well logged and commented. You can go back and see each changeset in detail (usually you have the full message from the email which the patch posted with an explanation).
Also the process of merging is much easier.
JA: With the recent release of 2.4.23, there were a number of VM related changes, including the removal of the OOM killer. Can you describe the intention and potential impact of these VM changes?
Marcelo Tosatti: The OOM killer can deadlock in some conditions, and can also kill tasks by mistake (accounting of shm/mlock'ed pages is not correct). Also it had problems with no swap setups.
For those who want the old OOM killer behaviour (which is useful if you expect to run hogs, etc), it will possible to select OOM-killer the next 2.4.x version.
JA: What conditions would cause the OOM killer to deadlock? Also, can you elaborate on why the OOM killer sometimes would kill tasks by mistake?
Marcelo Tosatti: It might select a TASK_UNINTERRUPTIBLE task (sleeping on NFS, for example), waiting on a NFS server which is down, which hangs the box for unlimited time.
A problem for servers is that OOM killer might kill big tasks (lets say, 3GB databases, etc) instead of smaller hogs (lets say, 1GB mozilla's). (that case the lack of the OOM killer will kill the right process).
The OOM killer can also detect the need of killing a process in cases with lots of mlock()'ed memory (which wont be swapped out) but still theres some swap free.
There are also some other corner cases which the OOM killer gets wrong (one of them related with small memory configurations).
JA: Will the optional OOM killer in 2.4.24 be identical to what was the default in 2.4.22?
Marcelo Tosatti: Yes.
JA: Historically the 2.4 VM has been a bit of a bumpy ride. How stable do you feel it is today?
Marcelo Tosatti: The 2.4.23 VM is pretty fast (fast in comparison with other 2.4's). I received a lot of successful reports.
The last VM changes in 2.4.23 also do a better job at HIGHMEM memory reclaiming/balancing.
There still a few problems with lots of HIGHMEM and some specific workloads, but thats pretty much all.
For most uses I think the VM has been very stable since 2.4.17/2.4.18.
JA: How much memory is considered "lots of HIGHMEM", and what are the remaining problems?
Marcelo Tosatti: More than 4GB and with lots of filesystem activity (using lots of files).
The inode reclaiming routine fails to free inodes which have pagecache data. With highmem that becomes a real problem. Andrea has a fix for this (inode_highmem from -aa tree)
JA: You recently announced your intention to put the 2.4 kernel into maintenance mode after the upcoming 2.4.24 release. What has lead to this decision?
Marcelo Tosatti: It's a timing issue. As I answer this question, 2.6.0 is out. Developers should concentrate on 2.6.0. It is stable for most users.
People who still use 2.4 will have their setups supported without any major modifications.
The same thing was done to 2.2 when 2.4 was getting into production.
JA: What are some of the main areas you'll be focusing on during the 2.4.24 release cycle?
Marcelo Tosatti: Merge the remaining "obviously safe" -aa (and other trees) fixes and make it sure it's stable (help tracking and booking of problems that show up in lkml or privately).
JA: What's else do you intend to merge from -aa?
Marcelo Tosatti: inode_highmem and maybe some other race fix.
JA: How long before we should expect to see 2.4.24?
Marcelo Tosatti: About mid-Feb 2004.
JA: How do you decide when it's time to release another version of the kernel?
Marcelo Tosatti: It depends on what are the modifications being done. We want the modifications to be well tested. For example, 2.4.21 had a big IDE update... that release took a lot of time to released.
If less modifications are done, the time to release a kernel can be smaller.
JA: Once 2.4.24 is released, you should have more time to focus on other things. How do you expect to spend this new found spare time?
Marcelo Tosatti: I want to get more involved with kernel development. I haven't had much time to do it since I started maintaining 2.4. I will also get more involved into the cyclades projects.
JA: What area of the kernel do you intend to get involved with?
Marcelo Tosatti: I'll try to get involved with the VM again (help bug chasing, etc), and try to help in general as I can.
There are some modifications from Cyclades (in the PPC architecture code) which I want to get merged.
JA: There recently was a lengthy debate on the Linux Kernel mailing list regarding whether XFS should be merged into the 2.4 kernel. Can you explain your reasons for not wanting to merge XFS?
Marcelo Tosatti: I did not want to merge XFS because I wasn't sure about the reliability of the VFS modifications.
Christoph Hellwig reviewed the patches which made me much more comfortable to merge them (the modifications got pretty straightforward after he and Nathan Scott from XFS agreed).
JA: Have there been any reported problems with XFS since it was merged?
Marcelo Tosatti: Yes, a problem was caused by a full filesystem in some cases IIRC. It's already been fixed.
JA: During the 2.4.23 release cycle, a bug was fixed in the do_brk() function. This bug was recently exploited in a high profile break-in of four Debian Project Linux servers. Why was 2.4.23 not released sooner when this bug was first fixed?
Marcelo Tosatti: When I first applied the fix (sent from Andrew Morton), I didn't realize it was an exploitable bug (I understood it could crash the box).
If I knew it I would have release the kernel a bit earlier.
JA: What reflections can you offer on SCO's recent allegations against the Linux kernel?
Marcelo Tosatti: I'm far from being a well informed person on this matter (I dont follow the case closely or anything), but my personal conclusions are that SCO does not threat Linux. They will loose in court.
JA: Have you had a chance to work with the new 2.6 kernel?
Marcelo Tosatti: I'm trying to get it to work on some of Cyclades boxes... (the kernel needs some modifications to work on them).
I'm also using 2.6 in my desktop.
JA: What do you feel are some of the more exciting new features in 2.6?
Marcelo Tosatti: The most exciting thing is the speed. It is really fast.
Also the infrastructure is much much better now (driver API, VM, IO, FS'es).
JA: What do you think will be some of the areas focused on during 2.7 development?
Marcelo Tosatti: It's hard to say, but I believe mostly VM/block IO (SCSI, IDE), networking...
JA: You've worked with Andrew Morton while maintaining the 2.4 kernel. How do you think he'll be as the 2.6 maintainer?
Marcelo Tosatti: I believe Andrew will be a great maintainer. He is very very skilled, and also knows how to have good relationships with people.
JA: How do you prefer to spend your time when you're not busy maintaining the 2.4 stable kernel?
Marcelo Tosatti: Well, I like to spend time with my wife, watch TV, read (computer and non computer related material :)), visit friends, etc.
JA: What types of non-computer books do you enjoy reading?
Marcelo Tosatti: Well there's not so much of a "type"... I usually read books that some friend suggests. I just started reading the bestseller "Stupid White Men".
JA: Is there anything else you'd like to add?
Marcelo Tosatti: Thanks for www.kerneltrap.org [3]!
JA: Thank you for taking the time to answer my questions, and for all the time you've put into maintaining the 2.4 stable kernel. I rely on this kernel day in and day out, and it has proven to be quite stable for me.
Related Links:
- Cyclades [4] (http://www.cyclades.com/ [5])
- Linux kernel source code [6] (http://kernel.org/ [7])