"Ok, I've been slacking on the -stable front for a bit here, and didn't realize how far behind I've gotten. Everyone has been sending patches in, which is great, but now we are facing a HUGE 114 patch release," began Greg Kroah-Hartman. He continued:

"As there's no real way that everyone can review all of these patches, I've decided to split them up into 6 different categories, and will be sending patches out in these categories for review. If people can just glance over the ones in the areas they care about, I would really appreciate it."

The stable review resulted in six stable 2.6.23.y releases. The first, 2.6.23.2, contained bug fixes for the core kernel code. 2.6.23.3 contained bug fixes for architecture specific issues. 2.6.23.4 contained bug fixes for the core networking and wireless code. 2.6.23.5 contained bug fixes for networking drivers. 2.6.23.6 contained bug fixes for non-networking drivers. 2.6.23.7 contained file system bug fixes. These releases were followed by 2.6.23.8 containing a couple security fixes.

"For the last release, I stated that I thought the 2.6.22.12 release would be the last one in the 2.6.22.y series. Since then, I've received a number of other patches that would be nice to have in the .22.y tree," explained Greg KH. He continued:

"So, for a while, I'll keep the 2.6.22.y tree open, doing new releases every once in a while as they accumulate. I do this, for no other than the selfish reason that I use it every day on my openSuSE 10.3 boxes as that is the kernel base that release is on :)"

Greg KH and Chris Wright have been maintaining a -stable 2.6.x.y patchset for the 2.6.x and 2.6.(x-1) kernels since March of 2005. 2.4 stable kernel maintainer Willy Tarreau has also maintained patches against the 2.6.20 branch since August of 2007, though noted that he'll switch to maintaining the stable 2.6.22 branch once Greg finishes. Adrian Bunk also continues to maintain a -stable 2.6.16 branch of the Linux kernel.

Greg KH and Chris Wright have been maintaining a -stable 2.6.x.y patchset for the 2.6.x and 2.6.(x-1) kernels since March of 2005. Thus, with the current stable release being 2.6.22, they maintain -stable patches for 2.6.22 and 2.6.21. 2.4 stable kernel maintainer Willy Tarreau noted the currently high patch rate in each of the 2.6 -stable trees and decided to maintain -stable patches against the 2.6.20 tree until things calm down. Adrian Bunk also continues to maintain a -stable 2.6.16 branch of the Linux kernel. Willy explained about his new 2.6.20 -stable patches:

"I proposed Chris and Greg to continue issuing a few more 2.6.20 releases during the time needed for 2.6.21 and 2.6.22 to show a significant drop in their patch rates, which hopefully will be just a matter of a few releases.

"My goal is *not* to do all the hard work they do, but just to backport from their patches those which are meaningful for 2.6.20. For this reason, 2.6.20 releases will always be slightly late and should not contain patches not merged in more recent releases."

In August 2006 Adrian Bunk took over maintainership of the 2.6.16.y stable kernel [story]. With the release of the 2.6.16.30-pre1 patch, concerns were expressed as to what makes a stable tree. The inclusion of new features led -stable maintainer Greg K-H to observe, "all of these patches seem like these are new features being backported to the 2.6.16.y kernel, which is not really allowed under the current -stable rules." Adrian responded, "they add support for additional hardware to the saa7134 driver. If you look at the actual diff there's not much that could cause any regression since nearly all of these changes don't change anything for the already supported cards." Greg cautioned, "if you want to accept new drivers and backports like this, I think you will find it very hard to determine what to say yes or no to in the future. It's the main problem that everyone who has tried to maintain a stable tree has run into, that is why we set up the -stable rules to be what they are for that very reason."

Willy Tarreau, maintainer of the 2.4 stable kernel [story] joined in with several others expressing concerns about keeping the 2.6.16 tree stable, "when I started the 2.4-hotfix tree nearly two years ago, I wanted to avoid merging driver changes as much as possible. And particularly, I avoided to add support for new hardware. The reason is very simple. I want to be able to guarantee that if 2.4.X works, then any 2.4.X.Y does too so that they can blindly upgrade." Adrian disagreed, "bugfixes causing regressions are much more likely than new hardware support adding regressions." He went on to note that he has two rules for accepting patches, first the patch must be in Linus' tree, and second he directly asks the patch authors and subsystem maintainers for feedback. "I do know that the only value of the 2.6.16 tree lies in a lack of regressions and act accordingly," Adrian added, "but I'm trying to do this in a pragmatic way."

With the release of the 2.6.16 Linux kernel, Adrian Bunk reiterated his previously debated intention of maintaining the 2.6.16.y kernel tree well into the future. The first 2.6.x.y release was 2.6.8.1 by Linus Torvalds [story], a quick one line fix for NFS. The idea was revisted a few months later in October of 2004 [story], but didn't actually gain momentum until March of 2005 [story] [story]. Beginning with the 2.6.11 kernel, the process was formalized with Greg KH and Chris Wright officially maintaining 2.6.x.y releases [story] until 2.6.(x+2) is released. For example, stable patches will be applied to the current 2.6.16.y kernel by Greg and Chris until 2.6.18 is released sometime well in the future.

Adrian's plan is to pick up the development of the 2.6.16.y kernel at that point, maintaining it much as the 2.4 kernel tree is is maintained [interview]. His intention is to maintain the tree as long is it is used and people contribute patches. The earlier debate on this idea was met with mixed reactions. At that time Greg KH cautioned, "the time and energy to do this for a long period of time is huge. If I were you, I would listen to the people who have and do maintain these kinds of kernels, it's not a simple job by any means."

At the July 2004 kernel summit, it was decided that the current 2.6 development process with teamwork between Andrew Morton [interview] and Linux creator Linus Torvalds was proving quite effective. The process involves using Andrew's test -mm tree [forum] as a staging area for patches prior to going into Linus' mainline tree [forum]. The system has allowed for continued evolution and new features in the 2.6 stable kernel, however it has also lead to a fair amount of discussion and debate [story]. Much of the concern is that with new features constantly being introduced, true stabilization may not be possible.

One theory presented on the lkml was that the process has changed because, "these days nobody wants to be a stable-release maintainer anymore. It's boring." 2.2 maintainer Alan Cox [story] disagreed, "that depends what kind of an engineer you are. Just as there are people who love standards body work and compliance testing/debugging there are people who care about stable trees." When asked if he was willing to maintain a stable 2.6.x kernel, Alan replied, "I'll do it if Linus wants". That is, while 2.6.10 is being developed, the suggestion is to continue to stabalize 2.6.9, releasing 2.6.9.1, 2.6.9.2, etc. And when 2.6.10 is released, to then focus on stabalizing it. Alan already maintains a 2.6-ac patchset [forum] which includes a growing number of bugfixes. However he notes that it is not intended to be all-inclusive, "the goal of -ac is to contain the stuff I personally consider important. A lot of the smaller bugfixes individually are fine but a 'complete set of bugfixes' turns into a large change set and then needs an entire validation and release cycle of its own."