Linux creator Linus Torvalds started a lengthy discussion on the lkml regarding release numbering for the Linux kernel. Some have complained about kernel stability with the new development model discussed back in mid-2004 [story] in which active development occurs in the "stable" 2.6 kernel. In his recent email, Linus explained, "the problem with major development trees like 2.4.x vs 2.5.x was that the release cycles were too long, and that people hated the back- and forward-porting. That said, it did serve a purpose - people kind of knew where they stood, even though we always ended up having to have big changes in the stable tree too, just to keep up with a changing landscape." His new proposal involves still using an even and odd numbering scheme, but on a smaller level. Thus, the upcoming 2.6.12 would be "stable" in that it should only contain bugfixes over 2.6.11. Then 2.6.13 would be more development oriented, including some larger changes. These larger changes would again stabalize in 2.6.14, and so on. He adds, "we'd still do the -rcX candidates as we go along in either case, so as a user you wouldn't even _need_ to know, but the numbering would be a rough guide to intentions. Ie I'd expect that distributions would always try to base their stuff off a 2.6.<even> release."

The lengthy discussion that followed was a collection of mixed reactions. Some liked the proposal, but others were confused as to what it was supposed to solve. Essentially the idea seems to be to get more people to test the kernel, as only with more testers can bugs be found. The current strategy of using a series of "-rc" kernels [story] is confusing to many as in most projects this indicates a "release candidate", or something thought to be stable, whereas with the Linux kernel an -rc release is frequently where the active development takes place. As the common user has come to realize this, the -rc kernels have gotten less testing. Linus says, "that's the whole point here, at least to me. I want to have people test things out, but it doesn't matter how many -rc kernels I'd do, it just won't happen. It's not a 'real release'." Andrew Morton [story]'s -mm tree is intended to weed out obvious errors with big changes before merging patches upstream in the mainline kernel, but again as it frequently proves less stable it tends to get less testing.

During the discussion, Linus notes that he's not unhappy with how 2.6 development is going, "I _do_ believe the current setup is working reasonably well." Instead, he suggested that he hopes to placate those that have requested additional stability, "although those people seem to not talk so much about 'it works' kind of stability, but literally a 'we can't keep up' kind of stability (ie at least a noticeable percentage of that group is not complaining about crashes, they are complaining about speed of development)." Read on for some interesting portions of this ongoing discussion. Or, follow it in real time.

From: Linus Torvalds [email blocked]
To: Kernel Mailing List [email blocked]
Subject: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 14:21:38 -0800 (PST)


This is an idea that has been brewing for some time: Andrew has mentioned
it a couple of times, I've talked to some people about it, and today Davem
sent a suggestion along similar lines to me for 2.6.12.

Namely that we could adopt the even/odd numbering scheme that we used to 
do on a minor number basis, and instead of dropping it entirely like we 
did, we could have just moved it to the release number, as an indication 
of what was the intent of the release.

The problem with major development trees like 2.4.x vs 2.5.x was that the 
release cycles were too long, and that people hated the back- and 
forward-porting. That said, it did serve a purpose - people kind of knew 
where they stood, even though we always ended up having to have big 
changes in the stable tree too, just to keep up with a changing landscape.

So the suggestion on the table would be to go back to even/odd, but do it 
at the "micro-level" of single releases, rather than make it a two- or 
three-year release cycle.

In this setup, all kernels would still be _stable_, in the sense that we
don't anticipate any real breakage (if we end up having to rip up so much
basic stuff that we have to break anything, we'd go back to the 2.7.x kind
of numbering scheme). So we should fear odd releases, but track them, to 
make sure that they are good (if you don't track them, and problems won't 
be fixed in the even version either)

But we'd basically have stricter concerns for an even release, and in
particular the plan would be that the diff files would alternate between
bigger ones (the 2.6.10->11 full diff was almost 5MB) and smaller ones (a
2.6.11->12 release would be a "stability only" thing, and hopefully the
diff file would be much smaller).

We'd still do the -rcX candidates as we go along in either case, so as a 
user you wouldn't even _need_ to know, but the numbering would be a rough 
guide to intentions. Ie I'd expect that distributions would always try to 
base their stuff off a 2.6.<even> release.

It seems like a sensible approach, and it's not like the 2.4.x vs 2.5.x
kind of even/odd thing didn't _work_, the problems really were an issue of

too big granularity making it hard for user and developers alike. So I see
this as a tweak of the "let's drop the notion althogether for now"  
decision, and just modify it to "even/odd is meaningful at all levels".

In other words, we'd have an increasing level of instability with an odd 
release number, depending on how long-term the instability is.

 - 2.6.<even>: even at all levels, aim for having had minimally intrusive 
   patches leading up to it (timeframe: a week or two)

with the odd numbers going like:

 - 2.6.<odd>: still a stable kernel, but accept bigger changes leading up 
   to it (timeframe: a month or two).
 - 2.<odd>.x: aim for big changes that may destabilize the kernel for 
   several releases (timeframe: a year or two)
 - <odd>.x.x: Linus went crazy, broke absolutely _everything_, and rewrote
   the kernel to be a microkernel using a special message-passing version 
   of Visual Basic. (timeframe: "we expect that he will be released from 
   the mental institution in a decade or two").

The reason I put a shorter timeframe on the "all-even" kernel is because I
don't want developers to be too itchy and sitting on stuff for too long if
they did something slightly bigger. In theory, the longer the better
there, but in practice this release numbering is still nothing but a hint
of the _intent_ of the developers - it's still not a guarantee of "we
fixed all bugs", and anybody who expects that (and tries to avoid all odd 
release entirely) is just setting himself up for not testing - and thus 
bugs.

Comments?

		Linus



From: Andrew Morton [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 14:58:59 -0800

Linus Torvalds [email blocked] wrote:
>
> The reason I put a shorter timeframe on the "all-even" kernel is because I
> don't want developers to be too itchy and sitting on stuff for too long if
> they did something slightly bigger.

If they're feeling itchy they should dig in and help fix the bugs in 2.6.<odd>.



From: Lars Marowsky-Bree [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 23:58:46 +0100

On 2005-03-02T14:21:38, Linus Torvalds [email blocked] wrote:

> We'd still do the -rcX candidates as we go along in either case, so as a 
> user you wouldn't even _need_ to know, but the numbering would be a rough 
> guide to intentions. Ie I'd expect that distributions would always try to 
> base their stuff off a 2.6.<even> release.

If the users wouldn't even have to know, why do it? Who will benefit
from this, then?

I think a better approach, and one which is already working out well in
practice, is to put "more intrusive" features into -mm first, and only
migrate them into 2.6.x when they have 'stabilized'.

This could be improved: _All_ new features have to go through -mm first
for a period (of whatever length) / one cycle. 2.6.x only directly picks
up "obvious" bugfixes, and a select set of features which have ripened
in -mm. 2.6.x-pre releases would then basically "only" clean up
integration bugs.

-mm would be the 'feature tree'. Of course, features which have matured
in other eligible trees might also work; the key point is the two-stage
approach and it doesn't matter whether the chaos stage has one or three
trees, as long as it's not more than that.

I think that would be natural extension of how things already work and
just tightens the process some.


(From a vendor perspective, this would mean we'd be safe picking up any
2.6.x tree + select choices from x+1-pre plus whatever we are force fed
by those who pay.)

If one wanted to get fancy, which I'm throwing in just to make everybody
lose the core point of the argument: One could associate "points" with
each feature / patch in -mm, based on an estimation of how
intrusive/well-tested/dangerous/heavenly that patch is, and mandate that
only 42 points per 2.6.x release are allowed.

Of course, one could also apply common sense. But, that's not as silly.
Or way more so, but less amusing than voting wars.

> Comments?

The numbering scheme is more confusing and unclear, and complexity is
the enemy of reliability.



Sincerely,
    Lars Marowsky-Brée [email blocked]

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business



From: Linus Torvalds [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 15:44:58 -0800 (PST)



On Wed, 2 Mar 2005, Lars Marowsky-Bree wrote:
> 
> If the users wouldn't even have to know, why do it? Who will benefit
> from this, then?

They don't _have_ to know. But both users and developers can take 
advantage of this to time their patches.

> I think a better approach, and one which is already working out well in
> practice, is to put "more intrusive" features into -mm first, and only
> migrate them into 2.6.x when they have 'stabilized'.

That wouldn't change. But we still have the issue of "they have to be 
released sometime". This makes it clear to everybody when to merge, and 
when to calm down.

		Linus


From: Dave Jones [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 19:32:59 -0500

On Wed, Mar 02, 2005 at 03:44:58PM -0800, Linus Torvalds wrote:

 > > I think a better approach, and one which is already working out well in
 > > practice, is to put "more intrusive" features into -mm first, and only
 > > migrate them into 2.6.x when they have 'stabilized'.
 > 
 > That wouldn't change. But we still have the issue of "they have to be 
 > released sometime". This makes it clear to everybody when to merge, and 
 > when to calm down.

So is the problem that people aren't listening when you say "lets slow down" ?
Why would this change things for people who obviously ignore what you say ? :)

I'll bet you'll still get flooded with "lets see if Linus takes this despite
what he said in his last announcement" patches if we moved to this model.

The only thing that would make a difference afaics, would be you putting
your foot down and just ignoring such submissions ?

		Dave



From: Russell King <rmk+lkml@arm.linux.org.uk>
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 23:06:34 +0000

On Wed, Mar 02, 2005 at 02:21:38PM -0800, Linus Torvalds wrote:
> In other words, we'd have an increasing level of instability with an odd 
> release number, depending on how long-term the instability is.
> 
>  - 2.6.<even>: even at all levels, aim for having had minimally intrusive 
>    patches leading up to it (timeframe: a week or two)
> 
> with the odd numbers going like:
> 
>  - 2.6.<odd>: still a stable kernel, but accept bigger changes leading up 
>    to it (timeframe: a month or two).
>  - 2.<odd>.x: aim for big changes that may destabilize the kernel for 
>    several releases (timeframe: a year or two)
>  - <odd>.x.x: Linus went crazy, broke absolutely _everything_, and rewrote
>    the kernel to be a microkernel using a special message-passing version 
>    of Visual Basic. (timeframe: "we expect that he will be released from 
>    the mental institution in a decade or two").

This sounds good, until you realise that some of us have been sitting
on about 30 patches for at least the last month, because we where
following your guidelines about the -rc's.  Things like adding support
for new ARM machines and other devices, dynamic tick support for ARM,
etc.

If I'm going to have to sit on this stuff for another month, it'll bit
rot rather badly, and I might as well throw away all these patches now
and ask people not to send stuff other than pure bug fixes.

A slightly bigger problem is that I'm in the middle of merging a large
chunk of these for you right now, and at least Andrew has already pulled
some of these from my BK repo.

If we are going to do this, can we start it post 2.6.12 please?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core



From: Dave Jones [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 18:22:54 -0500

On Wed, Mar 02, 2005 at 11:06:34PM +0000, Russell King wrote:
 > On Wed, Mar 02, 2005 at 02:21:38PM -0800, Linus Torvalds wrote:
 > > In other words, we'd have an increasing level of instability with an odd 
 > > release number, depending on how long-term the instability is.
 > > 
 > >  - 2.6.<even>: even at all levels, aim for having had minimally intrusive 
 > >    patches leading up to it (timeframe: a week or two)
 > > 
 > > with the odd numbers going like:
 > > 
 > >  - 2.6.<odd>: still a stable kernel, but accept bigger changes leading up 
 > >    to it (timeframe: a month or two).
 > >  - 2.<odd>.x: aim for big changes that may destabilize the kernel for 
 > >    several releases (timeframe: a year or two)
 > >  - <odd>.x.x: Linus went crazy, broke absolutely _everything_, and rewrote
 > >    the kernel to be a microkernel using a special message-passing version 
 > >    of Visual Basic. (timeframe: "we expect that he will be released from 
 > >    the mental institution in a decade or two").
 > 
 > This sounds good, until you realise that some of us have been sitting
 > on about 30 patches for at least the last month, because we where
 > following your guidelines about the -rc's.  Things like adding support
 > for new ARM machines and other devices, dynamic tick support for ARM,
 > etc.
 > 
 > If I'm going to have to sit on this stuff for another month, it'll bit
 > rot rather badly, and I might as well throw away all these patches now
 > and ask people not to send stuff other than pure bug fixes.

The fact that this new approach serialises the stable/devel lineation
whereas traditionally it was parallel (2.x.y/2.x+1.y) is going to be
a real pain for a lot of maintainers.

In short, instead of a single 'merge with linus tree', I'm now going to
need a 'merge with linus' and 'merge with linus next time' tree for every
tree I maintain.. It's not impossible to maintain, but its extra burden.
Burden which a lot of folks may consider not worth it.

		Dave



From: Jeff Garzik [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 02 Mar 2005 18:45:39 -0500

Russell King wrote:
> This sounds good, until you realise that some of us have been sitting
> on about 30 patches for at least the last month, because we where
> following your guidelines about the -rc's.  Things like adding support
> for new ARM machines and other devices, dynamic tick support for ARM,
> etc.

30?  Try 310 changesets, in my netdev-2.6 pending queue.

	Jeff



From: Linus Torvalds [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 16:00:46 -0800 (PST)



On Wed, 2 Mar 2005, Jeff Garzik wrote:
> 
> 30?  Try 310 changesets, in my netdev-2.6 pending queue.



Note that I don't think a 2.6.<even> would have problems with things like 
driver updates.

This was somewhat brought on (at least for me, dunno about Davem) by 
things like 4-level page tables etc stuff. I don't think most people even 
realized how _smoothly_ that thing seems to have gone, even if the ppc64 
people ended up having some really nasty debugging (and they came through 
with flying colors, but they probably didn't much enjoy the thing).

I would not keep regular driver updates from a 2.6.<even> thing. But I 
_would_ try to keep things like all the TSO pain, the 4-level page tables, 
and in general big merges that have been in CVS trees etc, and can't claim 
to be "lots of small stuff".

For example, there's some fundamental 16-bit PCMCIA cleanups pending in
the -mm tree, meaning that the kernel can work with PCMCIA cards without a
"cardmgr" deamon. That would be something that is _not_ appropriate in the
first stable version. That doesn't mean that individual PCMCIA device 
drivers could not get updated.

But hey, you may be right. It might just not be obvious enough which class 
any particular merge falls under.

		Linus



From: Greg KH [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 16:20:47 -0800

On Wed, Mar 02, 2005 at 04:00:46PM -0800, Linus Torvalds wrote:
> 
> On Wed, 2 Mar 2005, Jeff Garzik wrote:
> > 
> > 30?  Try 310 changesets, in my netdev-2.6 pending queue.
> 
> Note that I don't think a 2.6.<even> would have problems with things like 
> driver updates.
> 
> This was somewhat brought on (at least for me, dunno about Davem) by 
> things like 4-level page tables etc stuff. I don't think most people even 
> realized how _smoothly_ that thing seems to have gone, even if the ppc64 
> people ended up having some really nasty debugging (and they came through 
> with flying colors, but they probably didn't much enjoy the thing).

I think this statement proves that the current development situation is
working quite well.  The nasty breakage and details got worked out in
the -mm tree, and then flowed into your tree when they seemed sane.

> I would not keep regular driver updates from a 2.6.<even> thing. But I 
> _would_ try to keep things like all the TSO pain, the 4-level page tables, 
> and in general big merges that have been in CVS trees etc, and can't claim 
> to be "lots of small stuff".

So, any driver stuff is just fine?  Great, I don't have an issue with
your proposal then, as it wouldn't affect me that much :)

I do understand what you are trying to achieve here, people don't really
test the -rc releases as much as a "real" 2.6.11 release.  Getting a
week of testing and bugfix only type patches to then release a 2.6.12
makes a lot of sense.  For example, see all of the bug reports that came
out of the woodwork today on lkml from the 2.6.11 release...

But accepting 310 netdev patches, 250 USB, 50 PCI, 50 I2c, and 100 ALSA
patches in one week and expect the tree to stay "stable" might be a
pretty unreasonable thing to wish for...

thanks,

greg k-h



From: Linus Torvalds [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 17:15:36 -0800 (PST)



On Wed, 2 Mar 2005, Greg KH wrote:
> 
> I think this statement proves that the current development situation is
> working quite well.  The nasty breakage and details got worked out in
> the -mm tree, and then flowed into your tree when they seemed sane.

Actually, the breakage I was talking about got fixed in _my_ tree.

I'd love for the -mm tree to get more testing, but it doesn't. 

> So, any driver stuff is just fine?  Great, I don't have an issue with
> your proposal then, as it wouldn't affect me that much :)

I don't know about "any", but yeah.

> I do understand what you are trying to achieve here, people don't really
> test the -rc releases as much as a "real" 2.6.11 release.  Getting a
> week of testing and bugfix only type patches to then release a 2.6.12
> makes a lot of sense.  For example, see all of the bug reports that came
> out of the woodwork today on lkml from the 2.6.11 release...

A large part of it is psychological. On the other hand, it may be that
Neil is right and it would just mean that people wouldn't even test the
odd releases (..because they want to wait a couple of weeks for the even
one), so it may not actually end up helping much.

The thing is, I _do_ believe the current setup is working reasonably well.  
But I also do know that some people (a fairly small group, but anyway)  
seem to want an extra level of stability - although those people seem to
not talk so much about "it works" kind of stability, but literally a "we
can't keep up" kind of stability (ie at least a noticeable percentage of
that group is not complaining about crashes, they are complaining about
speed of development).

And I suspect that _anything_ I do won't make those people happy.

		Linus



From: Greg KH [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 00:19:58 -0800

On Wed, Mar 02, 2005 at 05:15:36PM -0800, Linus Torvalds wrote:
> The thing is, I _do_ believe the current setup is working reasonably well.  
> But I also do know that some people (a fairly small group, but anyway)  
> seem to want an extra level of stability - although those people seem to
> not talk so much about "it works" kind of stability, but literally a "we
> can't keep up" kind of stability (ie at least a noticeable percentage of
> that group is not complaining about crashes, they are complaining about
> speed of development).
> 
> And I suspect that _anything_ I do won't make those people happy.

That single sentence sums it up perfectly.  When I have given talks
about how our current development cycle works, and what's happening with
it, people just feel odd seeing all of this change happen and get upset
at it.  Perhaps it's because they never paid attention before, or that
they are new to Linux and like to believe that old-style development
models were somehow "better", and that they know we are doing something
wrong.

But when pressed about the issue of speed of development, rate of
change, feature increase, driver updates, and so on, no one else has any
clue of what to do.  They respond with, "but only put bugfixes into a
stable release."  My comeback is explaining how we handle lots of
different types of bugfixes, by api changes, real fixes, and driver
updates for new hardware.  Sometimes these cause other bugs to happen,
or just get shaken out where they were previously hiding (acpi is a
great example of this issue.)  In the end, they usually fall back on
muttering, "well, I'm just glad that I'm not a kernel developer..." and
back away.

Like I previously said, I think we're doing a great job.  The current
-mm staging area could use some more testers to help weed out the real
issues, and we could do "real" releases a bit faster than every 2 months
or so.  But other than that, we have adapted over the years to handle
this extremely high rate of change in a pretty sane manner.

So don't spend too much time worring that we need to make those types of
people happy.  As Jeff said, no other OS is accepting changes at such a
high rate.  But then again, no other OS supports more devices, on more
different processors.  So we must be doing something right :)

thanks,

greg k-h



From: Jeff Garzik [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 03 Mar 2005 03:38:22 -0500

Greg KH wrote:
> On Wed, Mar 02, 2005 at 05:15:36PM -0800, Linus Torvalds wrote:
> 
>>The thing is, I _do_ believe the current setup is working reasonably well.  
>>But I also do know that some people (a fairly small group, but anyway)  
>>seem to want an extra level of stability - although those people seem to
>>not talk so much about "it works" kind of stability, but literally a "we
>>can't keep up" kind of stability (ie at least a noticeable percentage of
>>that group is not complaining about crashes, they are complaining about
>>speed of development).
>>
>>And I suspect that _anything_ I do won't make those people happy.
> 
> 
> That single sentence sums it up perfectly.  When I have given talks
> about how our current development cycle works, and what's happening with
> it, people just feel odd seeing all of this change happen and get upset
> at it.  Perhaps it's because they never paid attention before, or that
> they are new to Linux and like to believe that old-style development
> models were somehow "better", and that they know we are doing something
> wrong.
> 
> But when pressed about the issue of speed of development, rate of
> change, feature increase, driver updates, and so on, no one else has any
> clue of what to do.  They respond with, "but only put bugfixes into a
> stable release."  My comeback is explaining how we handle lots of
> different types of bugfixes, by api changes, real fixes, and driver
> updates for new hardware.  Sometimes these cause other bugs to happen,
> or just get shaken out where they were previously hiding (acpi is a
> great example of this issue.)  In the end, they usually fall back on
> muttering, "well, I'm just glad that I'm not a kernel developer..." and
> back away.

The pertinent question for a point release (2.6.X.Y) would simply be 
"does a 2.6.11 user really need this fix?"


> Like I previously said, I think we're doing a great job.  The current
> -mm staging area could use some more testers to help weed out the real
> issues, and we could do "real" releases a bit faster than every 2 months
> or so.  But other than that, we have adapted over the years to handle
> this extremely high rate of change in a pretty sane manner.

I think Linus's "even/odd" proposal is an admission that 2.6.X releases 
need some important fixes after it hits kernel.org.

Otherwise 2.6.X is simply a constantly indeterminent state.

We need to serve users, not just make life easier for kernel developers ;-)

	Jeff


From: Greg KH [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 01:01:06 -0800

On Thu, Mar 03, 2005 at 03:38:22AM -0500, Jeff Garzik wrote:
> The pertinent question for a point release (2.6.X.Y) would simply be 
> "does a 2.6.11 user really need this fix?"

"need this fix bad enough now, or can it wait until 2.6.12?"

> >Like I previously said, I think we're doing a great job.  The current
> >-mm staging area could use some more testers to help weed out the real
> >issues, and we could do "real" releases a bit faster than every 2 months
> >or so.  But other than that, we have adapted over the years to handle
> >this extremely high rate of change in a pretty sane manner.
> 
> I think Linus's "even/odd" proposal is an admission that 2.6.X releases 
> need some important fixes after it hits kernel.org.

I agree.

> Otherwise 2.6.X is simply a constantly indeterminent state.

Heh, true, but all software is in that state :)

> We need to serve users, not just make life easier for kernel developers ;-)

Damm those pesky users.  Without them our life would be so much
easier...

greg k-h



From: Jeff Garzik [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 03 Mar 2005 04:18:13 -0500

Greg KH wrote:
> On Thu, Mar 03, 2005 at 03:38:22AM -0500, Jeff Garzik wrote:
> 
>>The pertinent question for a point release (2.6.X.Y) would simply be 
>>"does a 2.6.11 user really need this fix?"
> 
> 
> "need this fix bad enough now, or can it wait until 2.6.12?"
> 
> 
>>>Like I previously said, I think we're doing a great job.  The current
>>>-mm staging area could use some more testers to help weed out the real
>>>issues, and we could do "real" releases a bit faster than every 2 months
>>>or so.  But other than that, we have adapted over the years to handle
>>>this extremely high rate of change in a pretty sane manner.
>>
>>I think Linus's "even/odd" proposal is an admission that 2.6.X releases 
>>need some important fixes after it hits kernel.org.
> 
> 
> I agree.
> 
> 
>>Otherwise 2.6.X is simply a constantly indeterminent state.
> 
> 
> Heh, true, but all software is in that state :)
> 
> 
>>We need to serve users, not just make life easier for kernel developers ;-)
> 
> 
> Damm those pesky users.  Without them our life would be so much
> easier...


hehe :)

I do honestly worry that sometimes we lose sight of what best serves our 
users.

We need to not only produce a useful kernel, but also package it in a 
way that is useful to the direct consumers of the kernel:  distros 
[large and small] and power users.

What do those direct consumers really want?  What best serves them?

	Jeff



From: Andrew Morton [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 02:15:06 -0800

Jeff Garzik [email blocked] wrote:
>
> We need to not only produce a useful kernel, but also package it in a 
>  way that is useful to the direct consumers of the kernel:  distros 
>  [large and small] and power users.

This comes down to the question "what are we making"?  Is it an end
product, or is it a technology which can be turned into an end product? 
Because the two are different things.

I'd say that mainline kernel.org for the past couple of years has been a
technology, not a product.

We do put effort into productising that technology every couple of months
because that's what a (very important) minority of our users want, and
because it is good discipline, but productisation hasn't been the main
focus.

If we were to get serious with maintenance of 2.6.x.y streams then that is
a 100% productisation activity.  It's a very useful activity, and there is
demand for it.  But it is a very different activity.  And a lot of this
discussion has been getting these two activities confused.



But there's something else:

I would maintain that we're still fixing stuff faster than we're breaking
stuff.  If you look at the fixes which are going into the tree (and there
are a HUGE number of fixes), many of them are addressing problems which have
been there for a long time.

So as long as we remain in this state, we don't need to do anything.  The
technology gets closer to a product until we reach the stage where the
fixage rate equals the breakage rate.   And we're not there yet.

(It's nice that patches are called "fix the frobnozzle gadget", but this
analysis would be a lot easier if people would also label their patches
"break the frobnozzle gadget" when that's what they do.  Oh well).

So I'd suspect that on average, kernel releases are getting more stable. 
But the big big problem we have is that even though we fixed ten things for
each one thing we broke, those single breakages tend to be prominent, and
people get upset.  It's fairly bad PR that Dell Inspiron keyboards don't
work in 2.6.11, for example...

And people will incorrectly (and even wildly) generalise as a result of
such silly little isolated bugs.  We can wholly address such problems with
a 2.6.x.y productisation series.



And something else:

I don't think 2.2 and 2.4 models are applicable any more.  There are more
of us, we're better (and older) than we used to be, we're better paid (and
hence able to work more), our human processes are better and the tools are
better.  This all adds up to a qualitative shift in the rate and accuracy
of development.  We need to take this into account when thinking about
processes.

It's important to remember that all those kernel developers out there
*aren't going to stop typing*.  They're just going to keep on spewing out
near-production-quality code with the very reasonable expectation that
it'll become publically available in less than three years.  We need
processes which will allow that.




And another else:


Many people on this mailing list want a super-stable kernel as their first
(and sometimes only) priority (the product group).  But others have other
requirements: to make their code avaialble, or to get their hardware
supported, or to fix that scalability problem (the technology group).  The
product group's interests are in conflict with the technology group's. 

There will be no solution to this problem which is completely satisfactory
to either party.


>  What do those direct consumers really want?  What best serves them?

That's the primary question, of course.

Except I'd replace the term "consumers" with the more general term
"customers".  And I'd then say that, for example, the SGI Altix development
group is a customer of the kernel development team.

And how do we satisfy that customer?  By merging scary patches and hence
screwing your "consumers" ;)

oops.  Tension..



From: Wichert Akkerman [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 11:26:55 +0100

Previously Andrew Morton wrote:
> I'd say that mainline kernel.org for the past couple of years has been a
> technology, not a product.

If you consider mainline a technology and distributions your main users,
what is the use of a stable release every months or two months? No
distribution is going to updates its release that often. Looking at the
Debian kernel packages it took at least a month just to get a single
release ready for distro use. Needless to say, Debian (or any other
distro) is not going to go through that for every release.

We already saw that with 2.0, 2.2 and 2.4 kernels: distributions rarely
used the last mainline release but older released with (a sometimes
huge amount of) patches.

So continueing that thought pattern; why not go for something like 6
month release cycles? That seems to fit with a distro release cycles.

> So I'd suspect that on average, kernel releases are getting more stable. 
> But the big big problem we have is that even though we fixed ten things for
> each one thing we broke, those single breakages tend to be prominent, and
> people get upset.  It's fairly bad PR that Dell Inspiron keyboards don't
> work in 2.6.11, for example...

same for latitude keyboards after a resume I just discovered :(

Wichert.

-- 
Wichert Akkerman [email blocked]    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.



From: Andrew Morton [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 02:45:42 -0800

Wichert Akkerman [email blocked] wrote:
>
> Previously Andrew Morton wrote:
>  > I'd say that mainline kernel.org for the past couple of years has been a
>  > technology, not a product.
> 
>  If you consider mainline a technology and distributions your main users,
>  what is the use of a stable release every months or two months?

I don't know, because we've never tried it.  One can only speculate, and
I'd speculate that the kernel would end up pretty crappy because there
would never be a point in time when the thing was sufficiently usable.

The need to periodically lock things down and concentrate on stability and
generally get our act together every couple of months is good discipline
even if we're only making a technology, IMO.  And it gets more testers.

>  So continueing that thought pattern; why not go for something like 6
>  month release cycles? That seems to fit with a distro release cycles.

That could work.  The releases are slowing down as we spend longer wringing
out bugs.  Six months would be a bit excessive though.

But before solutions are discussed, we need to work out what the question was.

What are we trying to fix, and why?

>  > So I'd suspect that on average, kernel releases are getting more stable. 
>  > But the big big problem we have is that even though we fixed ten things for
>  > each one thing we broke, those single breakages tend to be prominent, and
>  > people get upset.  It's fairly bad PR that Dell Inspiron keyboards don't
>  > work in 2.6.11, for example...
> 
>  same for latitude keyboards after a resume I just discovered :(

You know what to do ;)



From: Jeff Garzik [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 02 Mar 2005 19:44:26 -0500

Linus Torvalds wrote:

> 
> On Wed, 2 Mar 2005, Jeff Garzik wrote:
> 
>>30?  Try 310 changesets, in my netdev-2.6 pending queue.
> 
> 
> Note that I don't think a 2.6.<even> would have problems with things like 
> driver updates.

Nah, I agree with DaveJ -- there are definitely "dev" portions when it 
comes to driver updates.

Judging from recent posting from Bart, it looks like he has an evil plot 
to merge the IDE driver with libata.  libata will also eventually 
[perhaps with Bart's changes?] make the SCSI portion optional, as I have 
long promised.  And it's getting other new and destabilizing features.

There will be other changes in SCSI and block too, which want staging... 
  Some of the stuff I've been putting off until "2.7" will be re-thought 
into something that appears in the on-going 2.6 series.

If you don't have driver stability, you don't have a useful kernel...

	Jeff



From: Dave Jones [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 19:27:33 -0500

On Wed, Mar 02, 2005 at 04:00:46PM -0800, Linus Torvalds wrote:

 > I would not keep regular driver updates from a 2.6.<even> thing. 

Then the notion of it being stable is bogus, given how many regressions
the last few kernels have brought in drivers.  Moving from 2.6.9 -> 2.6.10
broke ALSA, USB, parport, firewire, and countless other little bits and
pieces that users tend to notice.

The reason that things like 4-level page tables worked out so well
was that it was tested in -mm for however long, so by the time it got
to your tree, the silly bugs had already been shaken out.

Compare this to random-driver-update.  -mm testing is a valuable
proving ground, but its no panacea to stability. There's no guarantee
that someone with $affected_device even tried a -mm kernel.

For it to truly be a stable kernel, the only patches I'd expect to
drivers would be ones fixing blindingly obvious bugs. No cleanups.
No new functionality. I'd even question new hardware support if it
wasn't just a PCI ID addition.

		Dave



From: Andrew Morton [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 20:38:12 -0800

Dave Jones [email blocked] wrote:
>
> On Wed, Mar 02, 2005 at 04:00:46PM -0800, Linus Torvalds wrote:
> 
>  > I would not keep regular driver updates from a 2.6.<even> thing. 
> 
> Then the notion of it being stable is bogus, given how many regressions
> the last few kernels have brought in drivers.  Moving from 2.6.9 -> 2.6.10
> broke ALSA, USB, parport, firewire, and countless other little bits and
> pieces that users tend to notice.

Grump.  Have all these regressions received the appropriate level of
visibility on this mailing list?

I too am a little put off by the number of regressions which certain
(admittedly tricky) subsystems keep on introducing.  One does wonder how
careful people are being at the development stage.  And that's a thing
which we can surely fix.

For example, here's today's crop (so far):

include/linux/soundcard.h:195: warning: `_PATCHKEY' redefined
include/linux/awe_voice.h:33: warning: this is the location of the previous definition
drivers/i2c/chips/ds1337.c:60: `I2C_DRIVERID_DS1337' undeclared here (not in a function)
drivers/i2c/chips/ds1337.c:60: initializer element is not constant
drivers/i2c/chips/ds1337.c:60: (near initialization for `ds1337_driver.id')
drivers/i2c/chips/ds1337.c: In function `ds1337_get_datetime':
drivers/i2c/chips/ds1337.c:155: structure has no member named `id'
drivers/i2c/chips/ds1337.c: In function `ds1337_set_datetime':
drivers/i2c/chips/ds1337.c:206: structure has no member named `id'
drivers/i2c/chips/ds1337.c: In function `ds1337_detect':
drivers/i2c/chips/ds1337.c:333: structure has no member named `id'
drivers/i2c/chips/ds1337.c:343: structure has no member named `id'
make[3]: *** [drivers/i2c/chips/ds1337.o] Error 1
make[2]: *** [drivers/i2c/chips] Error 2
make[1]: *** [drivers/i2c] Error 2
make[1]: *** Waiting for unfinished jobs....
sound/pci/als4000.c: In function `snd_als4000_create_gameport':
sound/pci/als4000.c:572: warning: `r' might be used uninitialized in this function
drivers/char/watchdog/alim1535_wdt.c:319: warning: `ali_pci_tbl' defined but not used
drivers/char/sx.c:255: warning: `sx_pci_tbl' defined but not used
drivers/char/applicom.c:68: warning: `applicom_pci_tbl' defined but not used
make: *** [drivers] Error 2
make: *** Waiting for unfinished jobs....
sound/pci/trident/trident_main.c: In function `snd_trident_gameport_trigger':
sound/pci/trident/trident_main.c:3125: warning: `return' with a value, in function returning void

> For it to truly be a stable kernel, the only patches I'd expect to
> drivers would be ones fixing blindingly obvious bugs. No cleanups.
> No new functionality. I'd even question new hardware support if it
> wasn't just a PCI ID addition.

Agree.



From: Dave Jones [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Thu, 3 Mar 2005 00:21:01 -0500

On Wed, Mar 02, 2005 at 08:38:12PM -0800, Andrew Morton wrote:
 > Dave Jones [email blocked] wrote:
 > >
 > > On Wed, Mar 02, 2005 at 04:00:46PM -0800, Linus Torvalds wrote:
 > > 
 > >  > I would not keep regular driver updates from a 2.6.<even> thing. 
 > > 

 > > Then the notion of it being stable is bogus, given how many regressions
 > > the last few kernels have brought in drivers.  Moving from 2.6.9 -> 2.6.10
 > > broke ALSA, USB, parport, firewire, and countless other little bits and
 > > pieces that users tend to notice.
 > 
 > Grump.  Have all these regressions received the appropriate level of
 > visibility on this mailing list?

For the most part these things are usually known about by their upstream
authors.  To give an example: ALSA update in 2.6.10 broke sound for a
bunch of IBM thinkpads. As it turns out there are quite a lot of these
out there, so when I released a 2.6.10 update for Fedora, bugzilla lit
up like a christmas tree with "Hey, where'd my sound go?" reports.
(It was further confused by a load of other sound card problems, but
 thats another issue).  This got so out of control, Alan asked the ALSA
folks to take a look, and iirc Takashi figured out what had caused the
problem, and it got fixed in the ALSA folks tree, and subsequently, in 2.6.11rc.

Now, during all this time, there hadn't been any reports of this problem
at all on Linux-kernel. Not even from Linus' tree, let alone -mm.
Which amazes me given how widespread the problem was.

 > I too am a little put off by the number of regressions which certain
 > (admittedly tricky) subsystems keep on introducing.  One does wonder how
 > careful people are being at the development stage.  And that's a thing
 > which we can surely fix.
 > For example, here's today's crop (so far):
 > 
 > include/linux/soundcard.h:195: warning: `_PATCHKEY' redefined
 > include/linux/awe_voice.h:33: warning: this is the location of the previous definition

compile time regressions we should be able to nail down fairly easily.
(someone from OSDL is already doing compile stats and such on each release
 [too bad they're mostly incomprehensible to the casual viewer])
The bigger problem is runtime testing. If things aren't getting the
exposure they need, we're going to get screwed over by something or other
every time Linus bk pull's some random driver repo.

		Dave



From: Matt Mackall [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 15:52:07 -0800

On Wed, Mar 02, 2005 at 02:21:38PM -0800, Linus Torvalds wrote:
> 
> This is an idea that has been brewing for some time: Andrew has mentioned
> it a couple of times, I've talked to some people about it, and today Davem
> sent a suggestion along similar lines to me for 2.6.12.
> 
> Namely that we could adopt the even/odd numbering scheme that we used to 
> do on a minor number basis, and instead of dropping it entirely like we 
> did, we could have just moved it to the release number, as an indication 
> of what was the intent of the release.

One last plea for the 2.4 scheme:

 a) all the crazy stuff goes in 2.6.x-preN, which ends up being
    equivalent to 2.6.<odd> and friends in your scheme
 b) bugfixes only in 2.6.x-rcN, which ends up being equivalent to
    2.6.<even>-* in your scheme.
 c) 2.6.x is always 2.6.x-rc<last> with just a version number change[1]

This has some nice features:

 - alternates as rapidly as you want between stable and development
 - no brown paper bag bugs sneaking in between -rc<last> and 2.6.x 
 - 2.6.* is suitable for all users, 2.6.*-rc* is suitable for almost
   all users
 - it's already in use for 2.4 and people are happy with it

I _really_ don't want to explain to people that they don't want to use
2.6.13 because it's an odd number but that 2.4.31 is just fine (and so
is 2.6.9). Nor do I want to teach my ketchup tool the difference
between 2.6-stable and 2.6-unstable.

> The problem with major development trees like 2.4.x vs 2.5.x was that the 
> release cycles were too long, and that people hated the back- and 
> forward-porting. That said, it did serve a purpose - people kind of knew 
> where they stood, even though we always ended up having to have big 
> changes in the stable tree too, just to keep up with a changing landscape.

I think naming the interim releases -pre/-rc has done this admirably
for 2.4.

--
Mathematics is the supreme nostalgia of our time.



From: Jeff Garzik [email blocked]
Subject: workflow (was Re: RFD: Kernel release numbering)
Date: 	Wed, 02 Mar 2005 19:17:57 -0500

Other ideas <thinking out loud>...

I maintain my netdev-2.6 queue by creating a ton of "subject-specific" 
repositories locally,

> 8139cp/     bonding/  ieee80211/    mips/     sis900/    typhoon/
> 8139too/    e1000/    ixgb/         misc/     skge/      viro-iomap/
> 8139too-2/  ham/      janitor/      mv643xx/  sk_mca/    viro-iomap-orinoco/
> ALL/        ibmlana/  kill-iphase/  pcnet32/  smc91x/    viro-isa-ectomy/
> atmel/      ibmtr/    mii/          s2io/     sundance/  wifi/

and I pull all of these into a master repository netdev-2.6 (locally 
'ALL').  netdev-2.6, in turn, gets pulled into -mm when Andrew makes a 
new release.  When I am ready to push something upstream, I pull one or 
more repositories into my net-drivers-2.6 queue, which is essentially my 
"push to upstream" queue.

This scheme allows me to cherry-pick fixes and features.  If a critical 
fix comes along, it doesn't have to wait for the other less-critical 309 
changesets to go.


We could have a linus-pending-2.6 queue, managed by Linus, that 
functions in a similar manner to my netdev-2.6:

a) Have a monthly or weekly official release.  Could be automated, but 
probably not.

b) snapshots of linus-pending-2.6 function as a development tree, taking 
over some of the current -mm purpose...  -mm is just too massive and 
multi-purpose at this point

c) locally [or publicly?], Linus sorts patches and 'bk pulls' into 
specific repositories, as I do with netdev-2.6.  "sata", "sata-fixes", 
"net drivers", "net drivers - janitorial", etc. could be potential pull 
queues from me to Linus.

d) When features/fixes are ready to move from linus-pending-2.6 to 
linux-2.6 repository, Linus just does a pull+push locally.  Fixes would 
likely go immediately into linux-2.6, unless it needs staging in 
linus-pending-2.6 first.  Potentially destabilizing stuff stays longer 
in linus-pending-2.6.


Thus, Linus controls the patch flow (and stability/features) simply by 
choosing when to pull stuff from the on-going dev tree that he manages.

e) Andrew continues doing what he's been doing:  merging tons of 
patches, staging big features, etc.


So what does this accomplish, besides increasing the complexity of 
Linus's work?  What's the point?

1) Creates a structure to enable linux-2.6 to be the on-going stable 
tree, by creating an official on-going dev tree.

2) Release early, release often.  The proposal that started this thread 
does the opposite: it slows things down.

3) Makes it easier to manage the flow of changesets.

4) Takes pressure off -mm to be "everything under the sun."  Users are 
far more likely to test a tree that is blessed with Holy Penguin Pee, 
and builds on all architectures.  And this in turn gives -mm more freedom.

	Jeff, trying to think outside the box



From: Jeff Garzik [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 02 Mar 2005 21:21:24 -0500

David S. Miller wrote:
> On Wed, 02 Mar 2005 19:29:35 -0500
> Jeff Garzik [email blocked] wrote:
> 
> 
>>If the time between big merges increases, as with this proposal, then 
>>the distance between local dev trees and linux-2.6 increases.
>>
>>With that distance, breakages like the 64-bit resource struct stuff 
>>become more painful.
>>
>>I like my own "ongoing dev tree, ongoing stable tree" proposal a lot 
>>better.  But then, I'm biased :)
> 
> 
> The problem is people don't test until 2.6.whatever-final goes out.
> Nothing will change that.
> 
> And the day Linus releases we always get a pile of "missing MODULE_EXPORT()"
> type bug reports that are one liner fixes.  Those fixes will not be seen by
> users until the next 2.6.x rev comes out and right now that takes months
> which is rediculious for such simple fixes.
> 
> We're talking about a one week "calming" period to collect the brown paper
> bag fixes for a 2.6.${even} release, that's all.

If we want a calming period, we need to do development like 2.4.x is 
done today.  It's sane, understandable and it works.

2.6.x-pre: bugfixes and features
2.6.x-rc: bugfixes only


> All this "I have to hold onto my backlog longer, WAHHH!" arguments are bogus
> IMHO.  We're using a week of quiescence to fix the tree for users so they
> are happy whilst we work on the 2.6.${odd} interesting stuff :-)

If you think it will be only a week, you're deluding yourself.  It will 
stretch out to a month or longer, and the backlog problems will be real.


A calming period is fine.  But this even/odd mess is just silly.

	Jeff



From: Linus Torvalds [email blocked]
Subject: Re: RFD: Kernel release numbering
Date: 	Wed, 2 Mar 2005 19:37:44 -0800 (PST)



On Wed, 2 Mar 2005, Jeff Garzik wrote:
> 
> If we want a calming period, we need to do development like 2.4.x is 
> done today.  It's sane, understandable and it works.

No. It's insane, and the only reason it works is that 2.4.x is a totally
different animal. Namely it doesn't have the kind of active development AT
ALL any more. It _only_ has the "even" number kind of things, and quite 
frankly, even those are a lot less than 2.6.x has.

> 2.6.x-pre: bugfixes and features
> 2.6.x-rc: bugfixes only

And the reason it does _not_ work is that all the people we want testing 
sure as _hell_ won't be testing -rc versions.

That's the whole point here, at least to me. I want to have people test 
things out, but it doesn't matter how many -rc kernels I'd do, it just 
won't happen. It's not a "real release".

In contrast, making it a real release, and making it clear that it's a 
release in its own right, might actually get people to use it. 

Might. Maybe.

			Linus

• Archive of above thread
• KernelTrap interview with Andrew Morton
• KernelTrap interview with Dave Jones
• KernelTrap interview with Russell King