"Ok, I've been slacking on the -stable front for a bit here, and didn't realize how far behind I've gotten. Everyone has been sending patches in, which is great, but now we are facing a HUGE 114 patch release," began Greg Kroah-Hartman. He continued:

"As there's no real way that everyone can review all of these patches, I've decided to split them up into 6 different categories, and will be sending patches out in these categories for review. If people can just glance over the ones in the areas they care about, I would really appreciate it."

The stable review resulted in six stable 2.6.23.y releases. The first, 2.6.23.2, contained bug fixes for the core kernel code. 2.6.23.3 contained bug fixes for architecture specific issues. 2.6.23.4 contained bug fixes for the core networking and wireless code. 2.6.23.5 contained bug fixes for networking drivers. 2.6.23.6 contained bug fixes for non-networking drivers. 2.6.23.7 contained file system bug fixes. These releases were followed by 2.6.23.8 containing a couple security fixes.

Ingo Molnar sent a merge request to Linus Torvalds for the latest CFS fixes. CFS, the Completely Fair Scheduler, was merged into the mainline Linux kernel in July of 2007. It was first included in the 2.6.23 kernel, released in October of 2007. The scheduler appears to be quickly stabilizing, visible in the minimal assortment of fixes contained in the latest source code push. Ingo Molnar summarized the changes:

"There are two cross-subsystem groups of fixes: three commits that resolve a KVM build fix on !SMP - acked by Avi to go in via the scheduler git tree because it changes a central include file. The other one is a powerpc CPU time accounting regression fix from Paul Mackerras.

"The remaining 14 commits: one crash fix (only triggerable via the new control-groups filesystem), a delay-accounting regression fix, two performance regression fixes, a latency fix, two small scheduling-behavior regression fixes and seven cleanups."

In a recent discussion on the Linux Kernel mailing list, it was asked "which companies are helping develop the kernel?" One response pointed out that that a recent article on lwn.net summarized this information nicely. Others pointed to a paper maintained by Greg KH, who responded:

"Yeah, but my paper didn't really track companies very well. The lwn.net article is the best, and below is my version of who did things in 2.6.23. Note, the lack of a company is not an indicator that they did nothing, just that I could not easily determine someone worked for them. I'll try to send out my 'who are you working for' emails in a week or so to see if I can further categorize the 'unknowns'."

According to Greg's email, organizations that contributed more than 100 changesets to the recently released 2.6.23 kernel included: Red Hat with 827 changesets (11.7%), IBM with 557 changesets (7.9%), the Linux Foundation with 528 changesets (7.5%), Novell with 449 changesets (6.3%), Intel with 242 changesets (3.4%), Oracle with 158 changesets (2.2%), MIPS Technologies with 143 changesets (2%), Nokia with 133 changesets (1.9%), and NetApp with 119 changesets (1.7%). Greg noted that there were a total of 7,075 changesets from 992 developers working for 126 different employers. 843 of the changesets (11.9%) were contributed by individuals reporting no sponsor for their work.

Andrew Morton posted his first -mm patchset against the recently released 2.6.23 kernel, preparing for a big merge of patches bound for inclusion in the upcoming 2.6.24 kernel. He noted:

"I've been largely avoiding applying anything since rc8-mm2 in an attempt to stabilise things for the 2.6.23 merge.

"But that didn't stop all the subsystem maintainers from going nuts, with the usual accuracy. We're up to a 37MB diff now, but it seems to be working a bit better."

"As far as my testsystem goes, v2.6.23 beats v2.6.22.9 in sysbench," explained Ingo Molnar in response to a posting showing the opposite results. He referred to his own testing results and explained:

"As you can see it in the graph, v2.6.23 schedules much more consistently too. [ v2.6.22 has a small (but potentially statistically insignificant) edge at 4-6 clients, and CFS has a slightly better peak (which is statistically insignificant)."

Ingo noted that he was nuable to find information as to how the other benchmark was generated, "there are no .configs or other testing details at or around that URL that i could use to reproduce their result precisely, so at least a minimal bugreport would be nice." He then offered some tips on how sysbench works and some suggested tunings, "sysbench is a pretty 'batched' workload: it benefits most from batchy scheduling: the client doing as much work as it can, then server doing as much work as it can - and so on. The longer the client can work the more cache-efficient the workload is. Any round-trip to the server due to pesky preemption only blows up the cache footprint of the workload and gives lower throughput."

"Finally. Yeah, it got delayed, not because of any huge issues, but because of various bugfixes trickling in and causing me to reset my 'release clock' all the time. But it's out there now, and hopefully better for the wait," Linus Torvalds said announcing the 2.6.23 kernel. He noted few changes since the last release candidate, "not a whole lot of changes since -rc9, although there's a few updates to mips, sparc64 and blackfin in there. Ignoring those arch updates, there's basically a number of mostly one-liners (mostly in drivers, but there's some networking fixes and soem VFS/VM fixes there too)." Source level changes can be viewed via the gitweb interface. A nice overview of all changes can be found at Kernel Newbies. Linus went on to describe his plan going forward:

"I want this to be what people look at for a few days, but expect the x86 merge to go ahead after that. So far, all indications are still that it's going to be all smooth sailing, but hey, those indicators seem to always say that, and only after the fact do people notice any problems ;)"

A bug report filed by Ingo Molnar regarding a procfs crash in the recently released 2.6.23-rc9 kernel was quickly tracked down by Linus Torvalds as a compiler bug. The bug was ultimately determined to be from a compiler optimization generated with an older version of GCC. Ingo was skeptical at first, "it's 4.0.2. Not the latest & greatest but I've been using it for 2 years and this would be the first time it miscompiles a 32-bit kernel out of tens of thousands of successful kernel bootups." Linus replied, "I am 100% sure. I can look at the disassembly, and point to the fact that your Oops happens on code that is simply totally bogus." He continued on to offer an interesting review of the crash, explaining line by line what should have been generated versus what actually was, causing the crash. In the end, Ingo switched to a distribution compiled GCC 4.1.2 and confirmed that the crash went away, "so you are completely right, it's a compiler bug in 4.0.2."

During the thread, Linus suggested that the optimization made by the compiler wasn't "legal", to which Alan Cox retorted, "pedant: valid. Almost all optimizations are legal, nobody has yet written laws about compilers. Sorry but I'm forever fixing misuse of the word 'illegal' in printks, docs and the like and it gets annoying after a bit." Linus playfully responded, "heh. When I'm ruler of the universe, it *will* be illegal. I'm just getting a bit ahead of myself." When asked how long until he expected to be ruler, Linus added, "I'm working on it, I'm working on it. I'm just as frustrated as you are. It turns out to be a non-trivial problem."

"I said I was hoping that -rc8 was the last -rc, and I hate doing this, but we've had more changes since -rc8 than we had in -rc8. And while most of them are pretty trivial, I really couldn't face doing a 2.6.23 release and take the risk of some really stupid brown-paper-bag thing," Linus Torvalds said, announcing the 2.6.23-rc9 kernel. He added, "so there's a final -rc out there, and right now my plan is to make this series really short, and release 2.6.23 in a few days. So please do give it a last good testing, and holler about any issues you find!" He went on to warn developers that the first thing planned for 2.6.24 was to merge the unified x86 architecture:

"This is also a good time to warn about the fact that we're doing the x86 merge very soon (as in the next day or two) after 2.6.23 is out, so if you have pending patches for the next series that touch arch/i386 or x86-64, you should get in touch with Thomas Gleixner and Ingo Molnar, who are the keepers of the merge scripts, and will help you prepare..

"Doing it as early as possible in the 2.6.24-rc4 series (basically I'll do it first thing) will mean that we'll have the maximum amount of time to sort out any issues, and the thing is, Thomas and Ingo already have a tree ready to go, so people can check their work against that, and don't need to think that they have to do any fixups after it his *my* tree. It would be much better if everybody was just ready for it, and not taken by surprise."

With the official release of the 2.6.23 kernel expected any day now, Andrew Morton posted his -mm merge plans for the 2.6.24 kernel. The current Linux kernel development model is to open up the mainline kernel for significant merges during the two weeks following a major kernel release. Thus, during the two weeks following the imminent release of the 2.6.23 kernel, subsystem maintainers will push their latest trees to Linus' mainline tree. Andrew Morton will also push many of the patches he collects in his -mm tree to Linus' mainline tree during these two weeks, as detailed in his email. At the end of the merge window, 2.6.24-rc1 will be released and the stabilization process begins, though in reality significant merges also often slip in between -rc1 and -rc2. A series of -rc kernels will be released, eventually leading to a stable 2.6.24 kernel two or three months after the process started, and it all starts again.

Noting the approaching 2.6.24 merge window which will follow the upcoming release of the 2.6.23 kernel, MultiMedia Card (MMC) subsystem maintainer Pierre Ossman described what he plans to push upstream, "this release will probably be one of the biggest ones for the MMC layer so far. The major pieces are SDIO and SPI support, but there are several small nuggets as well." Regarding the new Secure Digital Input Output (SDIO) stack he noted, "gone are the days of having to rely on proprietary stacks for SDIO support in Linux. So no more spotty support for hosts and possible GPL problems. SDIO will now be a standard feature of Linux." He also described three working drivers already ported to the new stack.

Pierre went on to discuss the Serial Peripheral Interface (SPI) stack, "the second large feature is the fact that you can now use your SPI controllers for MMC, SD and SDIO. Yes, even SDIO works nicely over SPI. This means that a lot more systems can get storage and expansion I/O at basically the cost of a connector." He added, "David Brownell is currently marked as providing 'odd fixes' for the mmc_spi driver, but we could really use a proper maintainer. So if you have sufficient experience with Linux' SPI interface and the time, please raise your hand."

"Ok, I think I'm getting close to releasing a real 2.6.23," began Linus Torvalds in his release announcement for the eighth release candidate of the upcoming 2.6.23 kernel. "Things seem to have calmed down, and I think Thomas Gleixner may have found the suspend/resume regression that has dogged us for a while, so I'm feeling happy about things." Linus continued:

"Of course, me feeling happy is usually immediately followed by some nasty person finding new problems, but I'll just ignore that and enjoy the feeling anyway, however fleeting it may be.

"The shortlog really is pretty short, and I'm appending the diffstat at the end too in case anybody cares, but basically it's just a number of fairly small but real fixes, with some support for a few new chips to the sky2 network driver.."

"sched_yield() is not - and should not be - about 'recalculating the position in the scheduler queue' like you do now in CFS," Linus Torvalds stated in a discussion with Completely Fair Scheduler author Ingo Molnar, pointing to the man pages to back up his argument that sched_yield should instead move a thread to the end of its queue, adding, "quite frankly, the current CFS behaviour simply looks buggy. It should simply not move it to the 'right place' in the rbtree. It should move it *last*."

Ingo described how it worked with the pre-2.6.23 scheduler, "the O(1) implementation of yield() was pretty arbitrary: it did not move it last on the same priority level - it only did it within the active array. So expired tasks (such as CPU hogs) would come _after_ a yield()-ing task." He went on to compare this to the new process scheduler , "so the yield() implementation was so much tied to the data structures of the O(1) scheduler that it was impossible to fully emulate it in CFS. In CFS we dont have a per-nice-level rbtree, so we cannot move it dead last within the same priority group - but we can move it dead last in the whole tree. (then they'd be put even after nice +19 tasks.) People might complain about _that_." He also noted that this would change the behavior for some desktop applications that call sched_yield(), "there will be lots of regression reports about lost interactivity during load."

"Ahoy me laddies (and beauties)," Linux creator Linus Torvalds began, announcing the seventh release candidate for the upcoming 2.6.23 kernel, "time for the traditional 'Talk Like a Pirate Day' kernel release!" He noted, "now, last year we had a full release (2.6.18 was immortalized on TLAP-2006), but this year I'm chickening out, and we're just doing what is hopefully going to be the last -rc release for the 2.6.23 series." Full source changes can be viewed via the gitweb interface. Linus also offered a brief summary of the changes:

"I'm not including the diffstat, because it got blown up by the resurrection of the sk98lin driver - because skge that is supposed to supplant it doesn't handle some of the hardware. Oh well.

"Apart from that, we had some mips, powerpc and xtense updates, and various driver tweaks. Things like the USB autosuspend revert should make people happier, and some more clockevents fixes should help suspend/restore on i386."

Following Andrew Morton's recent comment, "this just isn't working any more," Miles Lane asked, "what can be done to reduce the huge number of build fixes required to release an MM tree?" Andrew jokingly replied, "my mind turns to cattle prods." Regarding the suggestion that he could publicly list the offenders he quipped, "I could name names, but it would look like 'grep @ MAINTAINERS' ;))" He continued to say, "I don't think much can be done about it, really," going on to explain:

"See, what I do is to merge probably hundreds of patches into the -mm-only part of the tree and then, after a few days, get down and compile-test it all, then fix it, then runtime test it all, then fix that. Because it is vastly more efficient to do all this work against hundreds of patches than it is to do it against one patch at a time, no?

"And guess what? All the other maintainers do the same thing: someone sends you a patch, it looks good, so you commit it. After you've committed a decent batch of patches, get in there and test it all. Problem is, I often will get in there and do all that testing before the subsystem-tree owner has done his testing."

A frustrated sounding Andrew Morton released the 2.6.23-rc6-mm1 kernel as "a 29MB diff against 2.6.23-rc6." Many patches are merged first into Andrew's -mm tree for testing before being pushed to Linus' mainline tree during the merge window. Andrew suggested that the -mm process wasn't working as well as it could:

"It took me over two solid days to get this lot compiling and booting on a few boxes. This required around ninety fixup patches and patch droppings. There are several bugs in here which I know of (details below) and presumably many more which I don't know of. I have to say that this just isn't working any more."