In an announcement for the 2.6.25.10 stable kernel, Greg KH noted, "it contains a number of assorted bugfixes all over the tree. And once again, any users of the 2.6.25 kernel series are STRONGLY encouraged to upgrade to this release." The emphasis on the word strongly led to a lengthy discussion about how security fixes are handled in the Linux Kernel. Linus Torvalds replied, "I personally consider security bugs to be just 'normal bugs'. I don't cover them up, but I also don't have any reason what-so-ever to think it's a good idea to track them and announce them as something special." Later in the thread he went on to explain, "one reason I refuse to bother with the whole security circus is that I think it glorifies - and thus encourages - the wrong behavior. It makes 'heroes' out of security people, as if the people who don't just fix normal bugs aren't as important. In fact, all the boring normal bugs are _way_ more important, just because there's a lot more of them. I don't think some spectacular security hole should be glorified or cared about as being any more 'special' than a random spectacular crash due to bad locking."

Theodore T'so pointed out that other developers had different beliefs about disclosure than Linus and referred to mailing lists such as the private security@ list described in the SecurityBugs documentation, originally created in early 2005. He then described Linus' stance, "if Linus finds out about a security bug, he will fix it and check it into the public git repository right away. But he's very honest in telling you that is what he will do --- so you can choose whether or not to include him in any disclosures that you might choose to make." Regarding whether Full Disclosure is the best policy, Ted highlighted the fact that the debate has been going on for several decades, "it is clear that we're not going settle this debate now, and certainly not on the Linux Kernel Mailing List." Later in the discussion, Linus offered a succinct summary of his viewpoint, "my responsibility is to do a good job. And not pander to the people who want to turn security into a media circus."

"As part of the Linux Foundation Technical board, we confront the issue of closed source Linux kernel modules all the time, and we wanted to do something that could be seen as a general 'public statement' about them that is easy to understand and point to when people have questions," began Greg KH, explaining, "so, after working on this for a while, and asking some of the other major contributors and maintainers of the kernel, what we have is below." a FAQ on the Linux Foundation website provides more background on the statement, which was undersigned by nearly 140 Linux kernel hackers. The statement reads:

"We, the undersigned Linux kernel developers, consider any closed-source Linux kernel module or driver to be harmful and undesirable. We have repeatedly found them to be detrimental to Linux users, businesses, and the greater Linux ecosystem. Such modules negate the openness, stability, flexibility, and maintainability of the Linux development model and shut their users off from the expertise of the Linux community. Vendors that provide closed-source kernel modules force their customers to give up key Linux advantages or choose new vendors. Therefore, in order to take full advantage of the cost savings and shared support benefits open source has to offer, we urge vendors to adopt a policy of supporting their customers on Linux with open-source kernel code."

"Oh great, not yet-another-kernel-tree, just what the world needs..." began Greg KH, continuing, "yes, this is an announcement of a new kernel tree, linux-staging." He explained:

"In a long and meandering thread with some of the other kernel developers a week or so ago, it came up that there is no single place for companies and developers to put their code for testing while it gets cleaned up for submission into the kernel tree. All of the different subsystems have trees, but they generally only want code that is about to go into this release, or the next one. For stuff that is farther off, there is no place to go. So, here's the tree for it."

In a readme created for the new tree, Greg adds, "the linux-staging tree was created to hold drivers and filesystems and other semi-major additions to the Linux kernel that are not ready to be merged at this point in time." He also requested that the new tree be included in Linux -next, leading Theodore Ts'o to ask, "does this mean that the nature of linux-next is changing? I thought the whole point of linux-next was only to have what would be pushed to Linus in the near future, so we could check for patch compatibility issues." Greg explained that he was hoping for an exception for his new -staging tree as it only includes whole new drivers and filesystems, not changes to existng features, "there is stuff that users can use to get hardware to work that currently is not supported on kernel.org kernels at all." As an example he noted, "there are 2 big network drivers in there that support a wide range of devices that some people would like to see working :)"

Tony Luck offered some statistics focused on the frequency of developers that only contribute to the Linux kernel one time, "I skimmed through looking for drive-by contributors (defined as someone who contributes to just one release and is then never heard from again)." Starting with the 2.6.11 kernel, he suggested the following numbers: "63 [developers contributed patches] in version 2.6.11 [and then were] never seen again, 148 in version 2.6.12, 128 in version 2.6.13, 92 in version 2.6.14, 96 in version 2.6.15, 122 in version 2.6.16, 137 in version 2.6.17, 140 in version 2.6.18, 135 in version 2.6.19, 95 in version 2.6.20, 136 in version 2.6.21, 153 in version 2.6.22, 179 in version 2.6.23, 179 in version 2.6.24, and 304 in version 2.6.25".

It was pointed out that the statistics were artificially inflated due to name misspellings and other variations, and that many of the listed people are actually still working with the Linux kernel. Greg KH added, "well, you do know that the distribution of all of our users are: 50% only contributed 1 patch; 25% contributed 2; 12% contributed 3; 6% contributed 4 and so on?" He went on to point out:

"Our curve is leveling out much better now though. For the whole 2.5 release, the top 30 people did over 80% of the work. Now, the top 30 people are doing 30% of the work. So it is getting much better, as long as we still continue to keep our massive rate of change that we have going, and huge number of developers, we should be fine. So this list doesn't necessarily mean anything is wrong, only that 50% are one-time contributors. And I think that shows we are easy to get a change into our tree from just about anyone, not that we are driving people away."

Discussing the latest breakage of the linux-next tree, Stephen Rothwell noted that the problem went unnoticed due to the arm tree not currently being included, "this is why I would have liked you to participate in the linux-next tree ...". Arm maintainer Russell King questioned the usefulness, saying, "linux-next will not give me anything which -mm isn't giving me. As I said in the discussion, linux-next value is _very_ small for me. Sorry but true." Several stepped in to offer some reasons that the linux-next tree is useful.

Andrew Morton noted, "putting arm into linux-next means that Stephen (and git) handle the merges rather than having me (and not-git) do it. Which helps me. I expect that linux-next will get a lot more cross-compilation testing than -mm. Which helps you." Greg KH added, "getting your stuff into linux-next would provide a public place for others to base off of, making it easier for them to send patches to you ensuring that they apply properly. Which in the end, will help others be able to contribute easier, and help you by getting patches you do not need to rebase yourself." Stephen Rothwell summarized the advantages for a maintainer:

"5 times a week your tree gets merged with lots of other code destined for Linus' next release. From this you get to find out about things in other trees that clash with yours. This tree gets built on several architectures for several configs (including arm). So you find out if other trees will break yours. I am happy to build more (basically all) the arm configs as I have offered before."

"To quote you a number of years ago: 'Linux is evolution, not intelligent design'," noted Greg KH, quoting Linux creator Linus Torvalds. Linus expanded on the statement, "evolution often does odd (and 'suboptimal') things exactly because it does incremental changes that DO NOT BREAK at any point." He continued, "in other words, exactly *because* evolution requires 'bisectability' (any non-viable point in between is a dead end by definition) and does things incrementally, it doesn't do big flips." When alternative examples in evolution were pointed out, Linus suggested that the kernel was much simpler than a mammal and more similar to bacteria:

"In bacteria (and viruses), duplication of DNA/RNA is a big cost of living in general, and as a result there is *much* less junk DNA. So in an evolutionary sense, it's much closer to what the kernel should have (with occasional duplication of code and interfaces to allow new functionality, but rather aggressive pruning of the excess baggage). In other words, all of these choices are a matter of 'balance'. In some areas, excess code is not a sufficient downside, and we keep even broken source code around with no actual function, 'just because' (or rather, because the cost of carrying it around is so small that nobody cares)."

"I re-ran some statistics the other day on our kernel development rate, and changed my formula after Andrew accused me of severely undercounting the rate of change," noted Greg KH during a discussion about the stability of the Linux kernel while undergoing significant changes. He continued, "turns out that as of 2.6.24-rc8 for the 2.6.24 kernel release we did: lines added per day, 4945; lines removed per day, 2006; lines modified per day, 1702". Greg added:

"And note, that is real stuff, not renames or file moves at all, git handles not reporting that. That's for the 99 days that it took to do 2.6.24-rc8 (I need to re-run the scripts now that 2.6.24 is out.) It's fricken scarily amazing that things are still working at all... Just something to make you all sleep well at night :)"

Prefacing a series of 196 patches, Greg KH noted, "due to the low level nature of these patches, and because they touch so many different parts of the kernel, a number of the subsystem maintainers have asked me to get them in first to make merging other trees easier." Linus Torvalds agreed and quickly merged the patches into his tree. Greg summarized the many changes:

"They can be broken down into these major areas: Documentation updates (language translations and fixes, as well as kobject and kset documentation updates.); major kset/kobject/ktype rework and fixes; struct bus_type has been reworked to now handle the lifetime rules properly, as the kobject is properly dynamic; struct driver has also been reworked, and now the lifetime issues are resolved; the block subsystem has been converted to use struct device now, and not 'raw' kobjects; nozomi driver is added; lots of class_device conversions to use struct device instead."

"Ok, I've been slacking on the -stable front for a bit here, and didn't realize how far behind I've gotten. Everyone has been sending patches in, which is great, but now we are facing a HUGE 114 patch release," began Greg Kroah-Hartman. He continued:

"As there's no real way that everyone can review all of these patches, I've decided to split them up into 6 different categories, and will be sending patches out in these categories for review. If people can just glance over the ones in the areas they care about, I would really appreciate it."

The stable review resulted in six stable 2.6.23.y releases. The first, 2.6.23.2, contained bug fixes for the core kernel code. 2.6.23.3 contained bug fixes for architecture specific issues. 2.6.23.4 contained bug fixes for the core networking and wireless code. 2.6.23.5 contained bug fixes for networking drivers. 2.6.23.6 contained bug fixes for non-networking drivers. 2.6.23.7 contained file system bug fixes. These releases were followed by 2.6.23.8 containing a couple security fixes.

"For the last release, I stated that I thought the 2.6.22.12 release would be the last one in the 2.6.22.y series. Since then, I've received a number of other patches that would be nice to have in the .22.y tree," explained Greg KH. He continued:

"So, for a while, I'll keep the 2.6.22.y tree open, doing new releases every once in a while as they accumulate. I do this, for no other than the selfish reason that I use it every day on my openSuSE 10.3 boxes as that is the kernel base that release is on :)"

Greg KH and Chris Wright have been maintaining a -stable 2.6.x.y patchset for the 2.6.x and 2.6.(x-1) kernels since March of 2005. 2.4 stable kernel maintainer Willy Tarreau has also maintained patches against the 2.6.20 branch since August of 2007, though noted that he'll switch to maintaining the stable 2.6.22 branch once Greg finishes. Adrian Bunk also continues to maintain a -stable 2.6.16 branch of the Linux kernel.

"This argument seems to start from the assumption that any externally maintained kernel code *can* get into the kernel, which doesn't stand up to reality. Once you admit that there is code which, for very good reasons, won't ever be accepted into the mainline kernel tree, what you are saying amounts to: 'Code that isn't fit to be included in the mainline kernel isn't fit to exist at all'," Tilman Schmidt argued during the ongoing debate about whether or not LSM should support modules.

Greg KH responded, "what kind of code is not accepted into the mainline kernel tree for good reasons? What are these reasons? What specific code are you talking about?" He pointed to a wiki page on the Linux Driver Project website and explained, "I'm trying to compile a list of all known external modules and drivers and work to get them included in the main kernel tree to help prevent these kinds of things."