This is starting to get beyond frustrating for me.
Yesterday, I spent the whole day bisecting boot failures

on my system due to the totally untested linux/bitops.h

optimization, which I fully analyzed and debugged.
Today, I had hoped that I could get some work done of my

own, but that's not the case.
Yet another bootup regression got added within the last 24

hours.
I don't mind fixing the regression or two during the merge

window but THIS IS ABSOLUTELY, FUCKING, REDICULIOUS!
The tree breaks every day, and it's becomming an extremely

non-fun environment to work in.
We need to slow down the merging, we need to review things

more, we need people to test their fucking changes!

--

Well, I must say I second that.
I'm not seeing regressions myself this time (well, except for the one that

Jiri fixed), but I did find a few of them during the post-2.6.24 merge window

and I wouldn't like to repeat that experience, so to speak.
IMO, the merge window is way too short for actually testing anything.  I rebuild

the kernel once or even twice a day and there's no way I can really test it.

I can only check if it breaks right away.  And if it does, there's no time to

find out what broke it before the next few hundreds of commits land on top of

that.
Thanks,

Rafael

--

On Wed, 30 Apr 2008 21:36:57 +0200
<jumps up and down>
There should be nothing in 2.6.x-rc1 which wasn't in 2.6.x-mm1!
_anything_ which appears in 2.6.x-rc1 and which wasn't in 2.6.x-mm1 was

snuck in too late (OK, apart from trivia and bugfixes).
If we decide that we need to fix the oh-shit-lets-slam-this-in-and-hope

problem then I expect we can do so, via fairly relible means.
But the first attempt at solving it should be to ask people to not do that.

--

The problem I see with both -mm and linux-next is that they tend to be

better at finding the "physical conflict" kind of issues (ie the merge

itself fails) than the "code looks ok but doesn't actually work" kind of

issue.
Why?
The tester base is simply too small.
Now, if *that* could be improved, that would be wonderful, but I'm not

seeing it as very likely.
I think we have fairly good penetration these days with the regular -git

tree, but I think that one is quite frankly a *lot* less scary than -mm or

-next are, and there it has been an absolutely huge boon to get the kernel

into the Fedora test-builds etc (and I _think_ Ubuntu and SuSE also

started something like that).
So I'm very pessimistic about getting a lot of test coverage before -rc1.
Maybe too pessimistic, who knows?
		Linus

--

First of all:

I 100% agree with Andrew that our biggest problems are in reviewing code

and resolving bugs, not in finding bugs (we already have far too many

unresolved bugs).
But although testing mustn't replace code reviews it is a great help,

especially for identifying regressions early.
Finding testers should actually be relatively easy since it doesn't

require much knowledge from the testers.
And it could even solve a second problem:
It could be a way for getting newbies into kernel development.
We actually do only rarely have tasks suitable as janitor tasks for

newbies, and the results of people who do neither know the kernel

nor know C running checkpatch on files in the kernel have already

been discussed extensively...
I'll try to do this:

- create some Wiki page

- get a mailing list at vger

- point newbies to this mailing list

- tell people there which kernels to test

- figure out and document stuff like how to bisect between -next kernels
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

On Thu, 1 May 2008 03:31:25 +0300
I would argue instead that we don't know which bugs to fix first.

We're never going to fix all bugs, and to be honest, that's ok.

As long as we fix the important bugs, we're doing really well.

And at least for the kerneloops.org reported issues, we're doing quite ok.
For me, 'important' is a combination of effect of the bug and the number of people

it'll hit. A compiler warning on parisc is less important than easy to trigger filesystem corruption

in ext3 that way; more people will hit it and the effect is more grave.
For oopses and WARN_ON()'s were getting to the hang of this now with kerneloops.org,

at least for the oopses that aren't really hard fatal. One thing I learned at least is that

lkml is a poor representation of what people actually hit; it's a very very selective

audience.

oopses/warnons are only a subset of the bugs of course... but still.
So there's a few things we (and you / janitors) can do over time to get better data on what issues

people hit:

1) Get automated collection of issues more wide spread. The wider our net the better we know which

   issues get hit a lot, and plain the more data we have on when things start, when they stop, etc etc.

   Especially if you get a lot of testers in your project, I'd like them to install the client for easy reporting

   of issues.

2) We should add more WARN_ON()s on "known bad" conditions. If it WARN_ON()'s, we can learn about it via

   the automated collection. And we can then do the statistics to figure out which ones happen a lot.

3) We need to get persistent-across-reboot oops saving going; there's some venues for this
--

That might be OK.
But our current status quo is not OK:
Check Rafael's regressions lists asking yourself

"How many regressions are older than two weeks?" 
The kernel Bugzilla curerntly knows about 212 open regression bugs.

(And many more have not made it into Bugzilla.)
We have unmaintained and de facto unmaintained parts of the kernel where 
No disagreement on this, its just a different issue than our bug fixing

problem.
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

On Thu, 1 May 2008 14:30:38 +0300
"ext4 doesn't compile on m68k".

YAWN.
Wrong question...

"How many bugs that a sizable portion of users will hit in reality are there?"
And how many people are hitting those issues? If a part of the kernel is really

important to enough people, there tends to be someone who stands up to either fix

the issue or start de-facto maintaining that part.

And yes I know there's parts where that doesn't hold. But to be honest, there's

not that many of them that have active development (and thus get the biggest
No it's not! Knowing earlier and better which bugs get hit is NOT different

--

"Kernel oops while running kernbench and tbench on powerpc" took more

than 2 months to get resolved, and we ship 2.6.25 with this regression.
Granted that compared to x86 there's not a sizable portion of users

crazy enough to run Linux on powerpc machines...
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

That was a very subtle bug that only showed up on one particular

powerpc machine.  I was not able to replicate it on any of the powerpc

machines I have here.  Nevertheless, we found it and we have a fix for

it.  I think that's an example of the process working. :)
Paul.

--

Was it even a regression in the classical sense of the word?  Seemed

more of a latent bug that was simply never triggered before.
josh
--

That's right.  The bug has been there basically forever (i.e. since

before 2.6.12-rc2 ;) and no-one has been able to trigger it reliably

before.
Paul.

--

But for users this is a recent regression since 2.6.24 worked

and 2.6.25 does not.
If this problem was on x86 Linus himself and some other core developers

would most likely have debugged this issue and Linus would have delayed

the release of 2.6.25 for getting it fixed there.
And stuff that "only showed up on one particular machine" often shows up

on many machines (we only know in hindsight) and the "one particular

machine" is often due to the fact that of the many machines that might

trigger a regression only one was used for testing this -rc kernel.
This not in any way meant against you personally, and due to the fact

that the powerpc port is among the better maintained parts of the kernel

this regression eventually got fixed, but in many other parts of the

kernel this would have been one more of the many regressions that were 
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

Totally and utterly immaterial.
If it's a timing-related bug, as far as developers are concerned, nothing

they did introduced the problem.
So anybody who think s that "process" should have caught it is just being

stupid. 
Adrian, you're one of the absolutely *worst* in the camp of "everything

should be perfect". You really need to realize that reality is messy, and

things cannot be pefect.
You also need to realize and *understand* that aiming for "good" is

actually much BETTER than trying to aim for "perfect".
Perfect is the enemy of good.
			Linus

--

So I would like to ask you what an user should do when facing what is

probably a timing-related bug, as it appears I have the bad luck

of hitting one.
See for example my comments after this one

http://bugzilla.kernel.org/show_bug.cgi?id=10117#c11
This same problem is still present with yesterday's git, and sometimes

it hangs without hpet=disable and sometimes it doesn't. (And never

with hpet=disable in the boot command line)
And when it hangs I can see only _one_ "Switched to high resolution mode

on CPU x" message before the hang point, and when it boots fine there

is always the two of them in sequence:
Switched to high resolution mode on CPU 1

Switched to high resolution mode on CPU 0
And using vga=6 or vga=0x0364 makes a difference in the probability

of hanging.
I am just waiting -rc1 to be released to send an email with my

problem again, as I am unable to debug this myself.

I think this is ok from my part, right?
--

Quite frankly, it will depend on the bug.
If it's *reliably* timing-related (which sounds crazy, but is not at all

unheard of), it can be reliably bisected down to some totally unrelated

commit that doesn't actually introduce the problem at all, but that

reliably turns it on or off.
That can be very misleading, and can cause us to basically revert a good

commit, only to not actually fix the bug (and possibly re-introduce the

bug that the reverted commit tried to fix).
But sometimes it gives us a clue where the timing problem is. But quite

frankly, that seems to be the exception rather than the rule.
There have been issues that literally seemed to depend on things like

cacheline placement etc, where changing config options for code that was

never actually even *run* would change timing just enough to show a bug

pseudo-reliably or not at all.
The good news is that those timing issues are really quite rare. 
Tha bad news is that when they happen, they are almost totally 
Hey, it may well be a HPET+NOHZ issue. But it could also be that HPET is 
.. and yeah, these kinds of really odd and obviously totally unrelated

issues are a sign of a bug that is either simply hardware instability or

very subtly timing-related.
The reason I mention hardware instability is that there really are bugs

that happen due to (for example) power supply instabilities. Brownouts

under heavy load have been causes of problems, but perhaps surprisingly,

so has _idle_ time thanks to sleep-states!
The latter is probably due to bad powr conditioning on the CPU power

lines, where the huge current swings (going at high CPU power to low, and

back again) not only have made soem motherboards "sing" (or "hum",

depending on frequency) but also causes voltage instability and then

the CPU crashes.
Am I saying that's the reason you see problems? Probably not. Most

instabilities really are due to kernel bugs. But hardware instabilities do 
Yes. You've been a good bug reporter, and...
[ message continues ]

It happens a bit before that because when it hangs it doesn't

print the above lines, and when it does not hang these lines are
Yes you are right. When I have luck and the boot succeeds my Sony laptop
A few days ago I found this message in lkml in reply to a hpet patch

http://lkml.org/lkml/2007/5/7/361 in which the reporter also had

a similar hang, which was cured by hpet=disable. 
So it is in my TODO list to try to check out if that patch is

in the current -git and whether it can be reverted somehow (I

added Venki to the Cc: now)
Thanks a lot for the answer!

--

It depends on whether we are HPET is being force detected based on the

chipset or whether it was exported by the BIOS in ACPI table.
If it was force enabled and above patch is having any effect, then you
In any case, off late there seems to be quite a few breakages that are

related to HPET/timer interrupts. One of them was on a system which has

HPET being exported by BIOS

http://bugzilla.kernel.org/show_bug.cgi?id=10409

And the other one where we are force enabling based on chipset

http://bugzilla.kernel.org/show_bug.cgi?id=10561
And then we have hangs once in a while reports by you, Roman and Mark

here

http://bugzilla.kernel.org/show_bug.cgi?id=10377

http://bugzilla.kernel.org/show_bug.cgi?id=10117
Thanks,

Venki

--

..
Yeah.  This particular bug first appeared when NOHZ & HPET were added.

Somebody once suggested it had something to do with an SMI interrupt

happening in the midst of HPET calibration or some such thing.
But nobody who works on the HPET code has ever shown more than a casual

interest in helping to track down and fix whatever the problem is.
Cheers

--

I said I was waiting for -rc1 to be released to send another email

about my HPET problem, but curiously with v2.6.26-rc1-6-gafa26be

my laptop did not hang after 30+ boots and counting. 
Somewhere between 2.6.25-07000-(something) and the above kernel

something happened which changed significantly the probability

of hanging during boot. 
I could not boot more than 3 times in

a row without hanging with kernels up to 2.6.25-07000 (approximately),

and now I am still booting v2.6.26-rc1-6-gafa26be a few times a day

and no hangs yet.
Yesterday I started a "reverse" bisection, trying to find which

commit "fixed" it, but I still didn't finish (but it is past

-7200).
Of course I am not sure if after the 100th boot the latest -git
Well, I would like to thank Venki for his effort because he even

answered some private emails from me about this issue and is

tracking the bugzillas about it.

--

..
My experience with this bug, since 2.6.20 or so, has been that it comes

and goes with even the most innocent change in the .config file,

like turning frame pointers on/off.
Cheers

--

I never actually saw a statement to that effect (i.e. that 2.6.24

worked) from Kamalesh.  I think people assumed that because he

reported it against version X that version X-1 worked, but we don't
If I had been able to replicate it, or if it had been seen on more

than one machine, I would probably have asked Linus to wait while we

fixed it.  
There's a risk management thing happening here.  Delaying a release is

a negative thing in itself, since it means that users have to wait

longer for the improvements we have made.  That has to be balanced

against the negative of some users seeing a regression.  It's not an

absolute, black-and-white kind of thing.  In this case, for a bug

being seen on only one machine, of a somewhat unusual configuration, I

considered it wasn't worth asking to delay the release.
Paul.

--

He reported it as
[BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc
No general disagreement on this.
And my example was not in any way meant against you - it's actually

unusual and positive that a bug that once got the attention of being

on the regression lists gets fixed later.
Even worse is the situation with regressions people run into when 
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

Precisely.  Cherry-picking a single example such as the 68k thing and then
Another fallacy which Arjan is pushing (even though he doesn't appear to

have realised it) is "all hardware is the same".
Well, it isn't.  And most of our bugs are hardware-specific.  So, I'd

venture, most of our bugs don't affect most people.  So, over time, by

Arjan's "important to enough people" observation we just get more and more

and more unfixed bugs.
And I believe this effect has been occurring.
And please stop regaling us with this kerneloops.org stuff.  It just isn't

very interesting, useful or representative when considering the whole

problem.  Very few kernel bugs result in a trace, and when they do they are

usually easy to fix and, because of this, they will get fixed, often

quickly.  I expect netdevwatchdogeth0transmittimedout.org would tell a

different story.
One thing which muddies all this up is that bug reporters vanish.  Over the

years I have sent thousands and thousands of ping emails to people who have

reported bugs via email, three to six months after the fact.  Some were

solved - maybe a fifth.  About the same proportion of reporters reply and

give some reason why they cannot work on the bug.  In the majorty of cases

people don't reply at all and I suspect they're in the same category of

cannot-work-on-the-bug.
And why can't they work on the bug?  Usually, because they found a

workaround.  People aren't going to spend months sitting in front of a

non-functional computer waiting for kernel developers to decide if their

machine is important enough to fix.  They will find a workaround.  They

will buy new hardware.  They will discover "noapic" (234000 google hits and

rising!).  They will swap it with a different machine.  They will switch to

a different distro which for some reason doesn't trigger the bug.  They

will use an older kernel.  They will switch to Solaris.  Etcetera.  People

are clever - they will find a way to get around it.
I figure that after a bug is reported w...
[ message continues ]

On Thu, 1 May 2008 08:49:19 -0700
no I'm pushing "some classes of hardware are much more popular/relevant
I did not say "most people". I believe "most people" aren't hitting

bugs right now (or there would be a lot more screaming).

What I do believe is that *within the bugs that hit*, even the hardware

specific ones, there's a clear prioritization by how many people hit
now that's a fallacy of your own.. if you care about that one, it's 1)

trivial to track and/or 2) could contain a WARN_ON_ONCE(), at which

point it's automatically tracked. (and more useful information I

suspect, since it suddenly has a full backtrace including driver info

in it)

By your argument we should work hard to make sure we're better at

creating traces for cases we detect something goes wrong.
if it's a hardware bug there's little we can do.

If it's a hardware specific bug, yeah then it becomes a function of how
Given that a normal PC has maybe 10 components...

yes we don't want bugcreep that affects common hardware over time.

At the same time, by your argument, a bug that hits a piece of hardware

of which 5 are made (or left on this planet) is equally important to
This statement is so rediculous and self contradicting to what you

said before that I'm not even going to respond to it.

--

"popular/relevant" is hard to define.
E.g. if we'd go after "popular" we should only keep architectures like

ARM and x86 and ditch architectures like ia64 and s390 that have puny

userbases.
If your "or have the hardware in general" is meant seriously you have to

convince people that ARM must become a very high priority.
No matter whether one supports your "there's a clear prioritization"

view or not it anyway doesn't currently work since the areas covered by

people testing -rc kernels don't even remotely map the most popular 
kerneloops.org catches the easiest to solve bugs (there's a trace) and

helps in getting them fixed.
That's a very good thing.
And if we get more bugs into this easy to resolve state that would be

even better.
But it's only a small part of the complete picture of incoming bug

reports.
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

So the question is if we have a thousand bugs which only affect one

person each, and 70 million Linux users, how much should we beat up

ourselves that 1,000 people can't use a particular version of the

Linux kernel, versus the 99.9% of the people for which the kernel

works just fine?
Sometimes, we can't make everyone happy.
At the recent Linux Collaboration Summit, we had a local user walk up

to a microphone, and loosely paraphrased, said, "WHINE WHINE WHINE

WHINE I have have a $30 DVD drive that doesn't work with Linux.  WHINE

WHINE WHINE WHINE WHINE What are *you* going to do to fix my problem?"
Some people like James responded very diplomatically, with "Well, you

have to understand, the developer might not have your hardware, and

there's a lot of broken out here, etc., etc."  What I wanted to tell

this user was, "Ask not what the Linux development community can do

for you.  Ask what *you* can do for Linux?"  Suppose this person had

filed a kernel bugzilla bug, and it was one of the hundreds or

thousands of non-handled bugs.  Sure, it's a tragedy that bugs pile

up.  But if they pile up because of crappy hardware, that's not a

major tragedy.  If we can figure out how to blacklist it, and move on,
Hey, in this particular case, if this user worked around the problem

by buying new hardware, it was probably the right solution.  As far as

we know we don't have a systematic problem where huge numbers DVD

drives aren't working, so if there are a few odd ball ones that are

out there, we just CAN'T self-flagellate ourselves that we're not
... and maybe we can't solve hardware bugs.  Or that crappy hardware

isn't worth holding back Linux development.  And I'm not sure ignoring

it is that horrible of a thing.  And in practice, if it's a hardware

bug in something which is very common, it *will* get noticed very

quickly and fixed.  But if it's in a hardware bug in some rare piece

of hardware, the user is going to have to either (a) help us fix it,

or (b) decide that his time is more ...
[ message continues ]

On Thu, 1 May 2008 13:24:34 -0400
Many, many of these are regressions.  If old-linux works on that

hardware then new-linux can too.
(still wants to know what we did 2-3 years ago which caused thousands of

people to have to resort to using noapic and other apic-related boot option

workarounds)
--

Forcing APIC even when the BIOS didn't support them.
-Andi
--

Perhaps 2-3 years ago more people started using more hardware that

implements APIC. ;-)
-- Steve
--

And actually, core kernel developers are best for writing new bugs.
Really, the way I started out learning how the kernel ticks was to go and

try to solve some bugs that I was seeing (this was years ago). I get

people asking that they want to learn to be a kernel developer and they

ask what new feature should they work on? Well, honestly, the last thing

a newbie kernel developer should be doing is writing new bugs. We need to

send them to a URL that lists all the known bugs and have them pick one,

any one, and have them solve it. This would be the best way to learn part

of the kernel.
I even find that I understand my own code better when I'm in the debugging

phase.
People here mention differnt places to look at code, and besides the

kerneloops.org I really don't even know where to look for bugs, because I

haven't seen a URL to point me to.
The next time someone asks me how to get started in kernel programming, I

would love to tell them to go and look here, and solve the bugs. I'm

guessing that I should just point them to:
  http://janitor.kernelnewbies.org/
and tell them to focus on real bugs (not just comments and such) to get

fixed if they really want to learn the kernel.
-- Steve
--

On Thu, 1 May 2008 12:38:23 -0400 (EDT)
bugzilla.kernel.org is, umm, improving.
It would be an intersting exercise for someone to spend a few days seeing

how many of the bugzilla reports they personally can reproduce.  I'd guess

"zero".  There's a lesson in that.
The problem with bugzilla will be that it will be hard to find reports

where the reporter will be able to work with you on the fix - we've let

them go cold.
The most fruitful place to find fixable bugs is linux-kernel.  People who

report bugs there are sufficiently motivated to have actually sent the

email and the bug is still recent, so they probably haven't done the

Solaris install yet.
--

Agreed.
Thanks,

Rafael

--

<boggle>
How about "a bug which we just added"?  One which is repeatable.

Repeatable by a tester who is prepared to work with us on resolving it.

Those bugs.
Rafael has a list of them.  We release kernels when that list still has tens of

unfixed regressions dating back up to a couple of months.
--

On Thu, 1 May 2008 01:13:46 -0700
I know he does. But I will still argue that if that is all we work from, and treat

all of those equally, we're doing the wrong thing.

I'm sorry, but I really do not consider "ext4 doesn't compile on m68k" which is

on that list to be as relevant as a "i915 drm driver crashes" bug which is among

us for a while and not on that list, just based on the total user base for either of those. 
Does that mean nobody should fix the m68k bug?

Someone who cares about m68k for sure should work on it, or if it's easy for an ext4 developer,

sure. But if the ext4 person has to spend 8 hours on it figuring cross compilers, I say

we're doing something very wrong here. (no offense to the m68k people, but there's just

a few of you; maybe I should have picked voyager instead)
Maybe that's a "boggle" for you; but for me that's symptomatic of where we are today:

We don't make (effective) prioritization decisions. Such decisions are hard, because it

effectively means telling people "I'm sorry but your bug is not yet important". That's

unpopular, especially if the reporter is very motivated on lkml. And it will involve a

certain amount of non-quantifiable judgement calls, which also means we won't always be

right. Another hard thing is that lkml is a very self-selective audience. A bug may be

reported three times there, but never hit otherwise, while another bug might not be reported

at all (or only once) while thousands and thousands of people are hitting it.
Not that we're doing all that bad, we ARE fixing the bugs (at least the oopses/warnings) that

are frequently hit. So I wouldn't blindly say we're doing a bad job at prioritizing. I would

rather say that if we focus only on what is left afterwards without doing a reality check,

we'll *always* have a negative view of quality, since there will *always* be bugs we don't

fix. Linux well over ten million users (much more if you count embedded devices).

A lot of them will have "standard" hardware, and a bunch of the...
[ message continues ]

On that note, I'd really like to see better binary availability of cross

compilers. While it's improved over the last few years mostly due to the

crossgcc stuff it's still a pain. Ideally, they would be available through

the distribution package manager even but failing that some dedicated place

on kernel.org with x86->lots and some of the more widely used other

combinations would quite definitely be good. Perhaps not really directly

relevant to this thread as such, but still good.
Andrew maintain{s,ed} a number of them at
http://userweb.kernel.org/~akpm/cross-compilers/
But as you see, most of the stuff there is really old again...
Rene

--

You're most welcome to help out Vegard to do this:
http://www.kernel.org/pub/tools/crosstool/

--

You could also use ct-ng:
http://ymorin.is-a-geek.org/dokuwiki/projects/crosstool
Works excellent for me :)
cu

--

---------------------------------------------------------------------

 Enrico Weigelt    ==   metux IT service - http://www.metux.de/

---------------------------------------------------------------------

 Please visit the OpenSource QM Taskforce:

 	http://wiki.metux.de/public/OpenSource_QM_Taskforce

 Patches / Fixes for a lot dozens of packages in dozens of versions:

	http://patches.metux.de/

---------------------------------------------------------------------

--

Ah, thanks, lovely, just new I see (and yes, I meant s/grossgcc/crosstool/).

Good thing. I'll check it out and see if there's anything to add.
Rene.

--

It's not that clear-cut, either. Something which manifests itself as a

build failure or an immediate test failure on m68k alone, might actually

turn out to cause subtle data corruption on other platforms.
You can't always know that it isn't important, just because it only

shows up in some esoteric circumstances. You only really know how

important it was _after_ you've fixed it.
That obviously doesn't help us to prioritise.
--

dwmw2
--

Ideally, you'd do an analysis first and then prioritize, based

on the severity of the bug, its exposure, how easy it is it fix,

etc.  If while doing that you already have a fix at hand, you're

almost done :)
Recursively, there's the problem of which bugs you analyze first.

I'm inclined to say that you want to analyze most if not all bug reports

in higher priority than working on fixing non-critical bug.
Benny

--

On Thu, 01 May 2008 13:42:44 +0100
absolutely. I'm not going to argue that prioritization is easy. Or

that we'll be able to get it right all the time.

--

And leave unfixed all the regressions introduced in earlier kernel versions

and known at the time of the release of that version but still present in

the current version? Not to mention all the other bugs reported by users of 
That can be true for not-so-recently introduced bugs too.
There are so many bugs out there and developers tend to focus on new ones

leaving a lot of others unattended, both important and not so important

ones.
Which ones should someone focus on? Maybe on the ones that someone (helped)

introduce him/herself. Maybe that should even sometimes be prioritized over

introducing new bugs^W^W^Wdoing new development.

--

<big_snip />
Hi folks,
what do you think about Gentoo's "bug-wrangler" concept ?

Maybe could do something similar:
An Tester group (which eg. should be the entry point for newbies),

is responsible for receiving bug reports from users (maybe even

distro maintainers who're not directly involved in kernel dev.).

They try to reproduce the bugs and find out as much as they can,

then file a report to the actual kernel devs (just critical bugs

are directly kicked to the devs with high priority). Maybe this

group could also keep users informed about fixes and give some

upgrade advise, etc.
This way we can build an good technical support (independent

from distributors ;-P), newbies can learn on the job and te

load on kernel devs is reduced, so they can better concentrate

on their core competences.
What do you think about this ?
cu

--

---------------------------------------------------------------------

 Enrico Weigelt    ==   metux IT service - http://www.metux.de/

---------------------------------------------------------------------

 Please visit the OpenSource QM Taskforce:

 	http://wiki.metux.de/public/OpenSource_QM_Taskforce

 Patches / Fixes for a lot dozens of packages in dozens of versions:

	http://patches.metux.de/

---------------------------------------------------------------------

--

Andrew already does more or less this.
The problems are:

- kernel bugs tend to very quickly reach the state where you need expert

  knowledge in some area, and there's definitely not much room for

  newbies in bug handling

- "try to reproduce the bugs" works for much software, but in the 
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

From: Adrian Bunk <bunk@kernel.org>
kernel-testers@vger.kernel.org has been created, feel free to

use it

--

Thanks  :-)

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

One thing is that we keep fragmenting the tester base by adding new

confidence levels: we now have -mm, -next, mainline -git, mainline -rc,

mainline release, stable, distro testing, and distro release (and some

distros even have aggressive versus conservative tracks.)  Furthermore,

thanks to craniorectal immersion on the part of graphics vendors, a lot

of users have to run proprietary drivers on their "main work" systems,

which means they can't even test newer releases even if they would dare.
This fragmentation is largely intentional, of course -- everyone can

pick a risk level appropriate for them -- but it does mean:
a) The lag for a patch to ride through the pipeline is pretty long.

b) The section of people who are going to use the more aggressive trees

for "real work" testing is going to be small.
	-hpa
--

And another problem is that often, it's hard to get good "real work" coverage

over the whole tree.  I just discovered an apparent borkage somewhere in

the networking/wireless area that seems to have gotten into Linus's tree

somewhere between 24-rc8 and 24-final, just because I haven't beaten on

my wireless card in the last few weeks, so I didn't notice a regression in

'ip link show' related to the rfkill switch...

Since I poke my head out of the foxhole every once in a while with a

relatively late-breaking bug report, I thought I should chime in...

Mr. Anvin has pretty much nailed it...
As the kernel development process has evolved, which "confidence level"

I select has evolved as well.  The thing that *hasn't* changed through

the years is, I tend to pick a "confidence level" that is appropriately

close to "mainline" and has an update release schedule roughly compatible

with my ability to keep up with it.  Specifically, if it takes me several

hours to download a patch set, apply it, build the new kernel, and test

on multiple platforms/architectures, then the update release schedule is

probably going to have to be no more often than twice a week if I'm going

to be at all interested in even trying to keep up with it.  In 2008, the

"-rcX" updates are a good fit.  In the not-too-distant past, keeping up

with 2.5.X.Y was no problem.
Yes, I realize I don't *have* to test every revision level in every

major tree, but I don't have to think about which one to pick for testing

if I can keep up with the update release schedule :-).
--

------------------------------------------------------------------------

Bob Tracy          |  "I was a beta tester for dirt.  They never did

rct@frus.com       |   get all the bugs out." - Steve McGrew on /.

------------------------------------------------------------------------

--

On Wed, 30 Apr 2008 13:31:08 -0700 (PDT)
Well.  We'll see.
linux-next is more than another-tree-to-test.  It is (or will be) a change

in our processes and culture.  For a start, subsystem maintainers can no

longer whack away at their own tree as if the rest of use don't exist.

They now have to be more mindful of merge issues.
Secondly, linux-next is more accessible than -mm: more releases, more

stable, better tested by he-who-releases it, available via git:// etc.  It

should be very easy for developers to do their weekly "does linux-next

boot" test.
Plus, of course, people who complain about merge-window breakage only to

find that the breakage was already in linux-next except they didn't test it

will not have a leg to stand on.
I feared that linux-next wouldn't work: that Stephen would stomp off in

disgust at all the crap people send at him.  But in fact it seems to be

going very well from that POV.
I get the impression that we're seeing very little non-Stephen testing of

linux-next at this stage.  I hope we can ramp that up a bit, initially by

having core developers doing at least some basic sanity testing.
linux-next does little to address our two largest (IMO) problems:

inadequate review and inadequate response to bug and regression reports.

But those problems are harder to fix..
--

Probably it would make sense also for distro vendors to make linux-next

snapshosts available in their development distro branches (redhat's

rawhide, opensuse's factory, etc), to make it easier to test by those

users who are willing to test if it works in their environment, but don't

want to compile kernels themselves.
--

Jiri Kosina
--

I try to test linux-next on a few SATA test boxes, but it's definitely 
Agreed...  any lead time on linux-next testing would be great.
	Jeff
--

Andrew, the latter thing is a very good point. For me personally, the fact

that -mm is not available via git is the major obstacle for trying your

tree more frequently than just a few times per year. How difficult it

would be to switch to git for you? I guess there are good reasons for still

using the source code management system from the last century; please

correct me if I'm wrong, but I believe that using a modern SCM system could
For busy (or lazy) people like myself, the big problem with linux-next are

the frequent merge breakages, when pulling the tree stops with "you are in

the middle of a merge conflict". Perhaps, there is a better way to resolve

this without just removing the whole repo and cloning it once again - this

is what I'm doing, please flame me for stupidity or ignorance if I simply

am not aware of some git feature that could be useful in such cases.
Finally, while the list is at it, I'd like to make another technical comment.

My development zoo is a pretty fast 4-way Xeon server, where I keep a handful

of trees, a few cross-toolchains, Qemu, etc. The network setup in our

organization is such that I can use git only over http from that server. This

cannot be changed, it's the company policy. In view of that, it's a pity that

quite a few tree owners don't make sure that http access to their trees works

(I added Ingo to the Cc: list in the hope that this will be corrected soon for

the x86 tree, which I am using quite extensively), and I have to use a much

slower machine (a two and a half year old laptop) for these trees. Please see

this:
<<<<<<<
[dmitri.vorobiev@amber ~]$ git clone http://www.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

Initialized empty Git repository in /home/dmitri.vorobiev/linux-2.6-x86/.git/

Getting alternates list for http://www.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

Also look at http://www.kernel.org/home/ftp/pub/scm/linux/kernel/git/torvalds/linux-2...

Getting pack list for http:...
[ message continues ]

On Thu, 01 May 2008 01:42:59 +0400
Every -mm release if available via git://, as described in the release

announcements.
The scripts which do this are a bit cantankerous but I believe they do

work.
<tests it>
Fatal, I expect.  A tool which manages source-code files is just the wrong
Really?  Doesn't Stephen handle all those problems?  It should be a clean
Don't know what to do about that, sorry.  An off-site git->http proxy might

work, but I doubt if anyone has written the code.
--

Would you mind using stgit? That you way have the queue patch

functionality, yet a simple git-push -f will send the whole

patch stack over to a repo (without the stgit bits that is),

leaving what looks like a regular tree with just lots of

recent commits. Does not even need extra scripts to do a
Indeed, assuming the remote is set up and you have a local branch,

`git reset --hard mm/master` after a fetch is the thing.

But be sure not to have any changed files.

--

Andrew Morton пишет:
But there is another solution, which I believe is straightforward: have the tree

maintainer set up his tree properly.
--

It should indeed be a clean fetch, but I wonder if Dmitri perhaps does a

"git pull" - which will do the fetch, but then try to _merge_ that fetched

state into whatever the last base Dmitri happened to have.
Dmitry: you cannot just "git pull" on linux-next, because each version of

linux-next is independent of the next one. What you should do is basically
	# Set this up just once..

	git remote add linux-next git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git
and then after that, you keep on just doing
	git fetch linux-next

	git checkout linux-next/master
which will get you the actual objects and check out the state of that

remote (and then you'll normally never be on a local branch on that tree,

git will end up using a so-called "detached head" for this).
IOW, you should never need to do any merges, because Stephen did all those

in linux-next already.
			Linus

--

Just to add some emphasis here - this is something that took me a long time to figure out, and since it is the pattern for dealing with the x86 trees and with the mm git tree and with linux-next, it would help if it were documented somewhere (not that I can imagine where).  Once you know it, it becomes obvious, but try staring at a merge conflict for a while trying to figure out what to do, and it gets frustrating.  I wonder if we can guess how many testers abandon the mm git tree or the linux-next tree because of this.
It might be nice if git supported a command like git-remote-help or something that would fetch a predefined help file from a remote tree that describes the workflow for that tree.
But at least with an extra reply to this mail, it might creep higher in the google search results when looking for merge conflicts with linux-next.
--

Kevin Winchester
--

Linus, thanks a lot for the detailed explanation. Indeed, it seems that I foolishly

tried to duplicate Stephen's work. In the future I'll do as you suggest here.
--

That "howto" should probably be added to the linux-next announcements...

(CC'ing Stephen)

--

This is already mentioned in the linux-next wiki

(http://linux.f-seidel.de/linux-next/pmwiki/) in the FAQ.  I will add a

link to the wiki to the announcements.
--=20

Cheers,

Stephen Rothwell                    sfr@canb.auug.org.au

http://www.canb.auug.org.au/~sfr/

Excellent idea. Thanks, Diego!
Dmitri

--

If this is still an issue of -next, I would say we won't get too much testers. I

gave up after first time I was attacked by that and got back to pure -mm.
I think greg-kh asked why this happens (Stephen rebases?), if you search

archives, I'm sure you'll find it.

--

From: Andrew Morton <akpm@linux-foundation.org>
This is all about positive and negative reinforcement.
The people who sit and git bisect their lives away to get the

regressions fixed need more positive reinforcement.  And the people

who stick these regressions into the tree need more negative

reinforcement.
The current way of dealing with folks who stick broken crud into the

tree results in zero change in behvaior.
People who insert the bum changes into the tree only really have one

core thing that they are sensitive to, their reputation.  That's why

there is an enormous reluctance to even suggest reverts, it looks bad

for them and it also makes more work for them in the end.
I guess what these folks are truly afraid of is that someone will

start tracking reverts and post their results in some presentation

at some big conference.  I say that would be a good thing.  To

be honest, hitting the revert button more aggressively and putting

the fear of being the "revert king" into everyone's minds might

really help with this problem.
Currently there is no sufficient negative pushback on people who

insert broken crud into the tree.  So it should be no surprise that it

continues.

--

David Miller wrote:

You will probably want to sort by "revert percentage" then.

The absolute number of reverts might make the biggest contributor

"revert king", even if his average patch quality is better than
Helge Hafting

--

What we need is not 'negative reinforcement'. That is just nasty, open

warfare between isolated parties, expressed in a politically correct

way.
The core problem is that every maintainer has his own subjective,

assymetric view and experience about this matter: to him his own tree is

almost problem-free and most problems are very easy to fix, while other

problems in other trees are nuisance that should never have been put

upstream.
Also, people get defensive when their regressions gets pointed out in

anything but the most respectful and casual manner.
For example, how on earth do i tell you that during the v2.6.24 merge

window, half of all x86 test-machines for me and others were broken

because they had no networking, for more than a week in a row? Are you

surprised about this (true) experience we had? Do you feel insulted? Do

you feel unfairly handled and slandered?
The same goes in the other direction as well - you were just hit by

scheduler tree related regressions that were only triggered on your

128-way sparc64, but not on our 64way x86 and smaller boxes.
The thing is, what we really need is more cooperation and earlier

integration - more people actually testing linux-next occasionally to

see how things will look like in the next merge window.
linux-next doing build tests is fine, but the nasty regressions that

will hit your box can only be solved if _you_ boot linux-next at least

once before the merge window opens. The regressions that will hit my box

can only be avoided if i test your tree.
hm? And can we please somehow talk about this without flaming each other

in the process?
	Ingo

--

Over time as patches succeed more I reduce testing so I can "get things done

faster".  Eventually I screw up, and get more cautious on checking.  It's a

dynamic balance.
With reduced review comes sloppier code.  If we can't increase review, we can

at least increase the penalty for screwing up when I do get caught.
If vger dropped all my emails for a week after I broke the kernel, I'd be far

more careful OR I'd find efficient ways to avoid doing that (like increasing

review, or automated testing).  Either way, it's a win.
But I'm sure everyone else is far more disciplined than I...

Rusty.

--

in more detail: any "negative reinforcement" should be on the

_technical_ level, i.e. when changes are handled - not at the broad tree

level.
Sure, there are exceptions, etc. - but by the time stuff goes upstream

it's too late and we've got to fix stuff instead of trying to push back

on each other.
by earlier integration (= linux-next) we can do the pushback much

earlier, in a much more granular, much more technical in a much less

personal way: "hey Ingo, your new sched-dizzy-blah patch broke stuff

here, zap it" or "hey Dave, that socket-foo rewrite just broke things

here, zap it".
git-revert _kind of_ makes that possible too, but people still feel too

personal about reverts - they take it as intrusion into their subsystem

and regard it as an attack against their competence as a maintainer.
and this is all so typical btw.: the most effective measure against

human warfare is for people to see each other and to talk to each other.
[ That's one reason why i am so worried about mailing list isolation.

  People get more distant, they mean less to each other, work less with

  each other => Linux suffers. I do accept that for some people lkml is

  simply too noisy - but i think the cure is worse than the disease. ]
	Ingo

--

From: Ingo Molnar <mingo@elte.hu>
Sure, and I'll provide some right here.
Ingo, let me know what I need to do to change your behavior in

situations like the one I'm about to describe, ok?
Today, you merged in this bogus "regression fix".
commit ae3a0064e6d69068b1c9fd075095da062430bda9

Author: Ingo Molnar <mingo@elte.hu>

Date:   Wed Apr 30 00:15:31 2008 +0200
    inlining: do not allow gcc below version 4 to optimize inlining
    fix the condition to match intention: always use the old inlining

    behavior on all gcc versions below 4.
    this should solve the UML build problem.
    Signed-off-by: Ingo Molnar <mingo@elte.hu>

    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Did you actually read the UML build failure report?
Adrian Bunk specifically stated that the UML build failure regression

occurs with GCC version 4.3
Next, did you test this regression fix?
Next, if you could not test this regression fix, did you wait

patiently for the bug reporter to validate your fix?  Adrian

responded that it didn't fix the problem, but that was after

you queued this up to Linus already.
This proves my main beef with you Ingo.  You're way too trigger happy,

you merge things in too quickly, without checks and without

verifications.
To an arbitrary person reading the commit logs, the above

looks like you fixed something, when you actually didn't fix

anything.
And let's address this specific inlining optimization and all the

fallout it's generating.  You said you merged this thing in because

you didn't want to "wait a year for such a useful feature."  In

hindsight, that's exactly what we should have done, waited until we

could sort out all of these issues.  Yes, even if it would take a

year.
Now we're forced to sort it out somehow, unless you can get beyond

your pride and revert the original change.

--

You got the facts wrong, it is even worse:
It was Ingo himself who reported this bug. [1]
Ingo managed to send an untested and not working patch for a bug he

reported himself...
cu

Adrian
BTW: I finally figured out what is behind the problems on UML, and this

     is not related to any recent kernel changes.

     Patch comes when I'm awake again.
[1] http://lkml.org/lkml/2008/4/26/151
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

the motivation of that fix wasnt UML - that was just an (indeed

incorrect) after-thought when i wrote up the commit log. The fix is

obviously right - although it doesnt fix UML.
it is wrong that it "doesnt fix anything". Look at the change itself:
- * Force always-inline if the user requests it so via the .config:

+ * Force always-inline if the user requests it so via the .config,

+ * or if gcc is too old:

  */

 #if !defined(CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING) || \

-    !defined(CONFIG_OPTIMIZE_INLINING) && (__GNUC__ >= 4)

+    !defined(CONFIG_OPTIMIZE_INLINING) || (__GNUC__ < 4)
before the change it was only possible to disable the optimization on

gcc 4 and above. The intended (and now implemented) condition is to only

change anything on gcc 4 and above. I.e. on gcc3x the config option has

no effect at all - and that's what we want.
	Ingo

--

On Thu, 1 May 2008 00:35:09 +0200
I'd question this.  People often seem pretty happy to yank their stuff out

of there - it relieves ongoing embarrassment and it relieves time pressure

- they can have another go and get it right at their leisure.
Of course, reverting is easy.  The hard part is often finding the thing

which needs to be reverted.
--

From: Ingo Molnar <mingo@elte.hu>
You keep saying this over and over again, but the powerpc folks hit

this stuff too.

--

Well, I think that some changes need some wider testing anyway.
They may be correct from the author's point of view and even from the knowledge

and point of view of the maintainer who takes them into his tree.  That's

because no one knows everything and it'll always be like this.
Still, with the current process such "suspicious" changes go in as parts of

large series of commits and need to be "rediscovered" by the affected testers

with the help of bisection.  Moreover, many changes of this kind may go in from

many different sources at the same time and that's really problematic.
In fact, so many changes go in at a time during a merge window, that we often

can't really say which of them causes the breakage observed by testers and

bisection, that IMO should really be a last-resort tool, is used on the main

debugging techinque.

--

That's very true IMHO and is the thing which has been progressively

appearing since we merge large amounts of code at once. In the "good

old days", something did not work, the first one to discover it could

quickly report it on LKML : "hey, my 128-way sparc64 does not boot

anymore, anybody has any clue", and another one immediately found

this mail (better signal/noise ratio on LKML at this time) and say

"oops, I suspect that change, try to revert it".
Now, it's close to impossible. Maintainers frequently ask for bisection,

in part because nobody knows what code is merged, and they have to pull

Linus' tree to know when their changes have been pulled. That may be

part of the "fun" aspect that Davem is seeing going away in exchange

for more administrative relations. But if we agree that nobody knows

all the changes, we must agree that we need tools to track them, and
Maybe we could slightly improve the process by releasing more often, but

based on topics. Small sets of minimally-overlapping topics would get

merged in each release, and other topics would only be allowed to pull

fixes. That way everybody still gets some work merged, everybody tests

and problems are more easily spotted.
I know this is in part what Andrew tries to do when proposing to

integrate trees, but maybe some approximate rules should be proposed

in order for developers to organize their works. This would begin

with announcing topics to be considered for next branch very early.

This would also make it more natural for developers to have creation

and bug-tracking phases.
Willy
--

What would this look like, notionally?  Say the releases were twice as

frequent with Stage A and Stage B.  How could the topic be grouped

into the stages?  Could bugfixes of any type be merged in either

window?  Would this only apply to "new" features, API changes, etc? or

would maintenance-type changes have to be assigned to a stage, too?
-chris

--

bug fixes are of course always possible, just that we limit important

changes, i.e. the ones which randomly break and that take a lot of time
willy
--

Yes, that's reasonable.
Thanks,

Rafael

--

git makes it easy to have many branches that get merged upstream, would it

really help much if these changes were initially done as seperate branches

and then merged in?
if so there are two ways to do this
have Ingo (and others) create a small forest of branches that get merged

into linux-next
have Ingo (and others) create a small forest of branches that get merged

into one 'please pull' branch that gets merged into linux-next
the second has the advantage that merge conflicts between the different

branches will be resolved before they go upstream, and there's less work

to be done upstream (as the upstream doesn't need to keep adding branches

to pull)
the first may have an advantage in terms of making the different branches 
there are always going to be cases where the problem can only be found by

bisecting it, but I agree that there seems to be a little too much

reliance on bisecting (but that was a heated topic a few weeks ago, let's

not re-hash it now)
David Lang

--

I'm not a frequent poster to this mailing list, but I do spend a good

portion of my life reading it. Please excuse me for expressing my very

personal opinion, but I thought you might probably be interested in a

detached view of the situation.
I think that many have guessed that I would like to talk about the attacks

to Ingo and backwards. Believe me, this fight looks childish, as it becomes

obvious that that went beyond purely technical disputes, which Linus is so

keen of rightfully writing about.
In no case am I implying any kind of offense, but I do believe that bad

emotions do hinder the community from advancing with the technical things.
Dmitri

--

... but that should also point at the trees through which the bugs are

introduced.
I mean, the maintainers should be more careful for what they take to their

trees and push upstream.  If that happens, they'll (hopefully) put some more

pressure on patch submitters.

--

Perhaps we should be clear and simple about what potential testers

should be running at any given point in time.  With -mm, linux-next,

linux-2.6, etc, as a newcomer I find it difficult to know where my

testing time and energy is best directed.
Is linux-next the right thing to be running at this point?  Is there a

need for testing in a particular tree (netdev, x86, etc)?
Cheers,

Dan
--

                     /--------------- - -  -  -   -   -

                     |  Dan Noe

                     |  http://isomerica.net/~dpn/

--

On Wed, 30 Apr 2008 16:47:00 -0400
-mm consists of the sum of
a) the ~80 subsytem maintainers trees (git and quilt)
b) the ~100 subsytem trees which are hosted only in -mm.
linux-next consists of only a)
Soon I shall remove a) from -mm and will replace it with linux-next (this

should be a no-op).
Later, I shall start feeding those 100 random subsystems into linux-next
yes.  85% of the code which goes into Linux goes via the ~80 subsystem

maintainers' trees and is (or should be) in linux-next.  The other 15%
No, please test the sum-of-all-trees in linux-next.  If you hit problems

then, as part of the problem resolving process a developer _might_ ask you

to test one tree specifically, but that would be a pretty unusual

circumstance.

--

Speaking of energy and time of a tester. I'd like to know where these resources

should be directed from the arch point of view. Once I had a plan to buy as

many arches as I could get and run a farm of test boxes 8-) But that's hard

because of various reasons (money, time, room, energy). What arches need more

attention? Which are forgotten? Which are going away? For example does buying

an alphaserver DS 20 (hey - it's cheap) and running tests on it makes sense

these days?
	Mariusz

--

A lot of bugs are not architecture specific. Or when they are architecture

specific they only affect some specific machines in that architecture.

But really a lot of bugs should happen on most architectures. Just focussing

on lots of boxes is not necessarily productive.
My recommendation would be to concentrate on deeper testing (more coverage)

on the architectures you have.
A interestig project for example would be to play with the kernel gcov patch that

was recently reposted (I hope it makes mainline eventually). Apply that patch,

run all the test suites and tests you usually run on your favourite test box

and check how much of the code that is compiled into your kernel was really tested

using the coverage information Then think: what additional tests can you do to get

more coverage?  Write tests then? Or just write descriptions on what is not tested

and send them to the list, as a project for others looking to contribute to the

kernel.
-Andi

--

Yes, there is some amount of bugs that I see only on specific architecture.

These which are reproducible or have an easy test case I do report to LKML, but

there are also bugs I see rarely or just once and they never come back and sometimes

as a bonus leave no trace - and these I ususaly don't report. Providing a test case
What I meant was one box per architecture, preferably an SMP one where possible - so

the number of required boxes is limited. This way instead of just cross-compiling

I could actually _run_ the kernel. On the other hand if some arch is close to be dead

and has no foreseable future then there is no point in testing it.
Also my thinking was that sometimes bugs from other (than x86) architectures can point to
Sounds like a plan - will look into that.
	Mariusz aka arch'aeologist ;)

--

On Thu, 1 May 2008 00:53:31 +0200
gee.
I think to a large extent this problem solves itself - the "more important"

architectures have more people using them, so they get more testing and

more immediate testing.
However there are gaps.  I'd say that arm is one of the more important

architectures, but many people who are interested in arm tend to shy away

from bleeding-edge kernels for various reasons.  Mainly because they have

real products to get out the door, rather than dinking around with mainline

kernel developement.  So testing bleeding-edge on some arm systems would be

good, I expect.
otoh, the platform we break most often is surely plain-old-PCs.  If it's

bugs you're looking for, I expect that dumpster-diving for as many

different PCs as you can and trying to get them to boot (let alone suspend

and resume!) would keep you entertained ;)
--

As both personally, and the policy of my employer we try and ensure we

can offer our customers at-least the previous 'stable' kernel release

and ensure that our development process tracks the kernel -rcX

candidates. We also run an autobuilder[1] which runs all -git releases

through an automated build (no auto-test yet) to ensure that we can

detect any build or configuration errors in the releases. 
ARM is a fast moving area due to the amount of sillicon vendors out

there who seem intent on doing their own thing, and often forking

hardware blocks they use during differing development branches. I

am currently looking at merging support for the S3C6400 (new) and

finishing S3C2443 (similar to 6400) and the S3C24A0... this means

that I have a lot of code to look through before each release and

having a stall will just keep the backlog building, making my job

a lot more difficult.
[1] http://armlinux.simtec.co.uk/kautobuild/
--

Ben (ben@fluff.org, http://www.fluff.org/)
  'a smiley only costs 4 bytes'

--

How bisectable is linux-next, BTW?

--

Each _individual_ release will be entirely bisectable, since it's all git

trees, and at no point does anything collapse individual commits together

like -mm does.
HOWEVER. 
Due to the way linux-next works, each individual release will be basically

unrelated to the previous one, so it gets a bit more exciting indeed when

you say "the last linux-next version worked for me, but the current one

does not".
Git can actually do this - you can make the previous (good) linux-next

version be one branch, and the not-directly-related next linux-next build

be another, and then "git bisect" will _technically_ work, but:
 - it will not necessarily be as efficient (because the linux-next trees

   will have re-done all the merges, so there will be new commits and

   patterns in between them)
 - but much more distressingly, if the individual git trees that got

   merged into linux-next were also using rebasing etc, now even all the

   *base* commits will be different, and saying that the old release was

   good tells you almost nothing about the new release!
   (The good news is that if only a couple of trees do that, the bisection

   information from the other trees that don't do it will still be valid

   and useful and help bisection)
 - also, while it's very easy for somebody who knows and understands git

   branches, it's technically still quite a bit more challenging than just

   following a single tree that never rebases (ie mine) and just bisecting

   within that one.
So yes, git bisect will work in linux-next, and the fundamental nature of

git-bisect will not change at all, but it's going to be a bit weaker

"between different versions" of linux-next than it would be for the normal

git tree that doesn't do the "merge different trees all over again" thing

that linux-next does.
		Linus

--

On Wed, 30 Apr 2008 23:30:20 +0200
don't know.  Fully, one hopes.
Laurent Riffard did a successful bisection last month; I don't see many

other signs on the linux-next list.
--

That is largely on purpose.
There's two choices:
 - have a longer and calmer merge window, spread out the joy, and have

   people test and fix their things during the merge window too. In other

   words, less black-and-white.
 - Really short merge window, and use the extra time *after* it to fix the

   issues.
and I've obviously gone for the latter. In fact, I'd personally like to

make it even shorter, because the problem with the long merge window can

be summed up very simply:
   Long merge windows don't work - because rather than test more, it just

   means that people will use them to make more changes!
So one of the major things about the short merge window is that it's

hopefully encouraging people to have things ready by the time the merge

window opens, because it's too late to do anything later.
And yes, we could have some other way of enforcing that - allow the merge

window to be longer, but have some other mechanism to make sure that I

only merge old code. 
In fact, I'd personally *love* to have a hard rule that says "I will only

pull from trees that were already 'done' by the time the window opened",

and we've been kind-of moving in that direction.
But that wish is counteracted by the fact that the merges themselves do

need some development, so expecting everything to be ready before-hand is

simply not realistic. 
Also, while I'd like trees to be ready when the window opens, at the same

time I do think that it's good to spread out some of it, and get *some*

basic testing - even if it's just a nightly build and a few tens of 
And really, that's all that we'd expect during the merge window. We want

to find the *obvious* problems - build issues, and the things that hit

everybody, but let's face it, the subtle ones will take time to find

regardless.
Then, the short merge window means that we have more time when we really

don't have big changes going in to find the subtle ones.
(And making the release cycle longer would *not*...
[ message continues ]

Having things ready by the time the merge window opens is difficult

when you don't know when the merge window is going to open.  OK, after

you release a -rc6 or -rc7, we know it's close, but it could still be

three weeks off at that point.  Or it could be tomorrow.
That's mitigated at the moment by having the merge window be two weeks

long.  So if you open the merge window at a point where I, or someone

downstream of me, thought we still had two weeks to go, we can hurry

up and try to get stuff finished within the first week and still get

it merged.
But if you made a really hard and fast rule that only stuff that is in

linux-next at the point where the merge window opens can be merged,

AND the point at which the merge window opens is unknown and

unpredictable within a period of about 4 weeks, then that makes it

really tough for those of us downstream of you to plan our work.
By the way, if you do want to make that rule, then there's a really

easy way to do it - just pull linux-next, and make that one pull be

the entire merge window. :)  But please give us at least a week's

notice that you're going to do that.
Paul.

--

Well, if the tree is ready, you shouldn't need to care ;)
I'm not going to pull linux-next, because I hate how it gets rebuilt every

time it gets done, so I would basically have to pick one at random, and

then that would be it.
I also do actually try to spread the early pulls out a _bit_, so that

if/when problems happen, there's some amount of information in the fact

that something started showing up between -git2 and -git3.
HOWEVER.
One thing that was discussed when linux-next was starting up was whether I

would maintain a next branch myself, that people could actually depend on

(unlike linux-next, which gets rebuilt).
And while I could do that for really core infrastructure changes, I really

would hate to see something like that become part of the flow - because

I'd hope things that really require it should be so rare that it's not

worth it for me to maintain a separate branch for it.
But there could be some kind of carrot here - maybe I could maintain a

"next" branch myself, not for core infrastructure, but for stuff where the

maintainer says "hey, I'm ready early, you can pull me into 'next'

already".
In other words, it wouldn't be "core infrastructure", it would simply be

stuff that you already know you'd send to me on the first day of the merge

window. And if by maintaining a "next" branch I could encourage people to

go early, _and_ let others perhaps build on it and sort out merge

conflicts (which you can't do well on linux-next, exactly because it's a

bit of a quick-sand and you cannot depend on merging the same order or

even the same base in the end), maybe me having a 'next' branch would be

worth it.
But it would have to be low-maintenance. Something I might open after

-rc4, say, and something where I'd expect people to only ask me to pull

_once_ (because they really are mostly ready, and can sort out the rest

after the merge window), and if they have no open regressions (again, the

"carrot" for good behaviour).
I'm not say...
[ message continues ]

linux-next is _supposed_ to be solely the stuff that is ready to be sent

to you upon window-open.
The only thing that isn't reliable are the commit ids -- and that's at

the request of a large majority of maintainers, who noted to Stephen R

that the branch he was pulling from them might get rebased -- thus

necessitating the daily tree regeneration.
So, I think a 'next' branch from you would open cans o worms:
- one more tree to test, and judging from linux-next and -mm it's tough

to get developers to test more than just upstream
- is the value of holy penguin pee great enough to overcome this

another-tree-to-test obstacle?
- opens all the debates about running parallel branches, such as, would

it be better to /branch/ for 2.6.X-rc, and then keep going full steam on

the trunk?  After all, the primary logic behind 2.6.X-rc is to only take

bug fixes, theoretically focusing developers more on that task.  But now

we are slowly undoing that logic, or at least openly admitting that has

been the reality all along.
	Jeff
--

That encourages developers to continue ignoring that stabilizing work.

The stall does have a side effect of refocussing them. A branch for -rc

and a monthly cycle would be interesting as it would mean that the

pushback for not fixing stability problems would be not getting you work

pulled for the main tree if you didn't fix the bugs first - and could be

both sufficient an incentive and not too vicious as it would be with a 2

month cycle.
Alan

--

Yes, the "stuff" may be supposed to be stable. But the trees feeding it

certainly are not. People are rebasing them etc, and it doesn't matter 
I do agree. And maybe I should have made it clear that I think it's worth

it to me only if it then means that the merge window can shrink.
If I'd have both a 'next' branch _and_ a full 2-week merge window, there's

no upside.
Btw, it wouldn't be another tree to test, since it would presumaby be what

'linux-next' starts out from - so it would purely be something that

doesn't have the constant re-merging of the more wild-and-crazy

'linux-next' tree.
			Linus

--

Personally I think the current process works reasonably well, though

as we should always try to improve it further...
I think you could branch at ~ rc3 (strictly critical fixes only from

this point). This way 'next' wouldn't be low-maintenance but the

release branch would be.
I.e., the merge window would open at ~ rc3. At 'final', the merge window

would probably be already closed :-)
Something like:

- 2.6.26-rc3: 2.6.27 merge window opens, 2.6.26 - fixes only

- 1 week later: no core changes for 2.6.27 except fixes (drivers only?)
2.6.26* would receive backports from 2.6.27 (cherry-picking? applying

on 2.6.26 and merging?).
The "no open regressions" rule would make sense certainly - unless in

a specific case agreed otherwise.
Perhaps if needed you could let other people do the final release
Shorter cycle is the big upside.
Perhaps we could start branching later at first - say at 2.6.26-rc5,

and see how does it work.

--

Krzysztof Halasa

--

Yep, that sounds pretty interesting. But It would be better to start something

like ,,slow merge window'' (explained below) around -rc4 where things really

slow down (or used to).
The idea of ,,slow merge window'' would look like:

	- merge only *obvious* (long awaiting) changes;

	- merge stuff (fixes) which comes to -rc releases;

	- merge non-core changes from -mm;
-Jacek

--

That's a unique and interesting idea...
	Jeff
--

Full ack.
Especially if there was some kind of "pre-merge linux-next freeze" where

people (arch maintainers, kernel testers) would be actively invited to do

pre-merge testing.
During that period only changes that fix reported issues (be it build issues

or regressions) would be allowed:

- either a revert of the problematic commit

- or a targeted fix
This could even hugely improve the bisectability of mainline after the merge

as such changes could be merged/rebased into the subsystem tree _before_

Linus pulls them into mainline.
Currently I avoid -next and -mm and I also don't do any merge window

testing. Why? Too much flux, too many issues, too much energy required.

But if there was some sort of pre-merge call for testing of an identifiable

and relatively stable tree, I would definitely participate in that and be

willing to spend time to bisect the hell out of any issues I'd find.
Cheers,

FJP

--

And what do you think is happening _after_ the merge window closes, when

we're supposed to be fixing bugs?  People work on new code.  And, in fact, they
How about, instead, putting limits on the amount of stuff that's going to be
Well, and when's the time for fixing bugs?  Surely not during the merge window

and also not after that, because otherwise people won't be ready for the next
Exactly.  Moreover, the code is now being merged at a pace that makes it
Sorry to say that, but I don't think this is realistic.  What happens after the merge

window is people go and develop new stuff.  They look at the already merged

code only if they have to.  Also, there are a _few_ people testing the kernel

carefully enough to see the more subtle problems, let alone debugging and
My point is, given the width of the merge windown, there's too much stuff

going in during it.  As far as I'm concerned, the window can be a week long

or whatever, but let's make fewer commits over a unit of time.
Thanks,

Rafael

--

That's not correct.  People work on new code before, during, and after 
To be ready for the next merge window just means to know which code is

sufficiently reviewed and tested, and to have it queued up and if

necessary synchronized with other pending code.

--

Stefan Richter

-=====-==--- -=-= ----=

http://arcgraph.de/sr/

--

I'm not quite sure if really all of them do.  Well, I should have said "some
Of course it _should_ mean that, but the fact is unreviewed and untested

patches are pushed to Linus, at least from time to time.  [Even some known

broken patches were pushed to Linus in the past, but we can't prevent that from

happening by any process changes.]
Thanks,

Rafael

--

Oh, I agree. But at that point, the issue you brought up - of testing and

then having the code change under you wildly - has at least gone away.
From a testing standpoint, the *developers* aren't ever even the main

issue. Yes, we get test coverage that way too, but we should really aim

for getting most of the non-obvious issues from the user community, and

not primarily from developers.
So the whole point of the merge window is *not* to have developers testing

their code during the six subsequent weeks, but to have *users* able to

use -rc1 and report issues!
That's why the distro "testing" trees are so important. And that's why 
I'm not following that logic. 
A single merge will bring in easily thousands of commits. It doesn't

matter if the merge window is a day or a week or two weeks, the merge will

be one event.
And there's no way to avoid the fact that during the merge window, we will

get something on the order of ten thousand commits (eg 2.6.24->25-rc1 was

9629 commits).
So your "fewer commits over a unit of time" doesn't make sense. We have

those ten thousand commits. They need to go in. They cannot take forever.

Ergo, you *will* have a thousand commits a day during the merge window.
We can spread it out a bit (and I do to some degree), but in many ways

that is just going to be more painful. So it's actually easier if we can

get about half of the merges done early, so that people like Andrew then

has at least most of the base set for him by the first few days of the

merge window.
So here's the math: 3,500 commits per month. That's just the *average*

speed, it's sometimes more. And we *cannot* merge them continuously,

because we need to have a stabler period for testing. And remember: those

3,500 commits don't stop happening just because they aren't merged. You

should think of them as a constant pressure.
So 3,500 commits per month, but with a stable period (that is *longer*

than the merge window) that means that the merge wind...
[ message continues ]

That's correct, but since developers are already working on new code at that

point, the bug reports in fact distract them and make them go back to the "old"

stuff, recall why they did that particular changes etc.  As a result, the

developers often do not take the bug reports seriously enough, especially if

they do not finger the "guilty" change.  That, in turn, makes the users believe
Well, do we _have_ _to_ take that much?  I know we _can_, but is this really
Oh, yes it does.  Equally well you could say that having brakes in a car

didn't make sense, even if you could drive it as fast as the engine allowed
Surely, they don't, but maybe they don't have to.
You can technically handle merging even more, but what about quality?  Do we

have a quality assurance process in place?  If we do, what is it?  How is it

able to handle the 3500 commits a week?  Assuming it is, will it be able to

handle more and what's the limit?
IMO, there has to be a limit somewhere, or we will end up in a spiral driving

everybody mad.
Thanks,

Rafael

--

not really, if patches are produced at a rate of 1000/week and you decide

to only accept 2000 of them this month, a month later you have 6000

patches to deal with. history has shown that developers do not stop

developing if their patches are not accepted, they just fork and go their

own way.
David Lang
--

Well, I think you know how TCP works.  The sender can only send as much

data as the receiver lets it, no matter how much data there are to send.

I'm thinking about an analogous approach.
If the developers who produce those patches know in advance about the rate

limit and are promised to be treated fairly, they should be able to organize
That's mostly when they feel that they are treated unfairly.
OTOH, insisting that your patches should be merged at the same rate that you're

able to develop them is unreasonable to me.
Thanks,

Rafael

--

We cannot control who develops what.
When someone wants some feature or wants to get Linux running on his

hardware he will always develop the code.
We can only control what we merge.
And the main rationale for the 2.6 development model was that we do no

longer want distributions to ship kernels with insane amounts of 
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

To be exact, we control what we merge and when.  There's no rule saying that

every patch has to be merged as soon as it appears to be ready for merging,
This was an argument agaist starting a separate development branch in analogy

with 2.5, IIRC, and I agree with that.
Still, I think we don't need to merge patches at the current rate and it might

help improve their overall quality if we didn't.  Of course, the latter is only

a speculation, although it's based on my experience.
Thanks,

Rafael

--

What currently gets applied to the kernel are between two and three

million lines changed per year.
We can discuss when and how to apply them.
But unless we want to create an evergrowing backlog we have to change

roughly 200.000 lines per month on average.
Even with higher quality criteria that might result in some code not 
See above - what do you want to do if we'd merge less and have a backlog

of let's say one million lines to change after one year, much of it

already in distribution kernels?
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

Well, I'm feeling that's what Linus is tryig to say too. :-)
I, for one, don't really want to cope with a situation I don't feel comfortable

in, because in the long run that leads to growing frustration.  It seems pretty

obvious to me that people generally get more and more frustrated with the

current development process and it will have to be addressed somehow anyway.
If there's a problem, and I think that there really _is_ one, we should at

least try to _address_ it instead of just trying to duck it.
Thanks,

Rafael

--

they will make the patches bigger to get the changes in a smaller number 
it's not nessasarily the individuals that fork, it's the distros who want

to include the fixes and other changes that the individuals that create

the fork.
David Lang

--

Is that really bad?  Isn't that effectively equivalent to "increased testing of

earlier intergrations"?
-chris

--

not if there are so many changes that the testing isn't really relavent to

mainline.
not if the changes don't get into mainline.
look at the mess of the distro kernels in the 2.5 and earlier days. having

them maintain a large body of patches didn't work for them or for the

mainline kernel.
David Lang

--

Exactly. 
I do think Rafael's TCP analogy is somewhat germane, but it misses the

point that the longer the queue gets, the *worse* the quality gets. It

gets worse because the queued-up patches don't actually get tested any

more during their queueing, and because everybody else who isn't

intimately involved with production of said patches just gets *less*

inclined to look at big patch-queue than a small one.
So having a long queue and trying to manage it (by some kind of negative

feedback) is counter-productive, because by the time that situation

happens, you're basically screwed already.
That's what we largely had with the Xen merge, for example. A lot of the

code had been around for basically _forever_, and the people involved in

reviewing it got really tired of it, and there was no way in *hell* a new

person would ever start reviewing the huge backlog. Once it is massive,

it's just too massive.
So trying to push back from the destination is really painful. It's also

aggravating for everybody else. When people were complaining about me not

scaling (remember those flame-wars? Now the complaint is basically the

reverse), it was very painful for everybody, and most of all me. 
So I really really hope that if we need throttling (and I do want to point

out that I'm not entirely sure we do - I think the issue is not "number of

commits", but "quality of code", and I do _not_ agree that the two are

directly related in any way), it should be source-based.
Trying to make sure that the source throttles, and not by making

developers feel unproductive. And quite frankly, most things that throttle

the source are of the annoying and non-productive kind. The classic source

throttle tends to be to make it very "expensive" to do development, by

introducing various barriers.
The barriers are usually "you need to have <n> other people look at it",

or "you need to pass this five-hour test-suite", and almost invariably,

the big issue is not code quality, b...
[ message continues ]

Heh.  The Xen code in the kernel now is a complete rewrite, with only

trace elements from the original patchset.  And yes, that's partly

because the original patches were unreviewable.
    J

--

Do you want me to stop merging your code?
Do you think anybody else does?
Any suggestions on how to convince people that their code is not worth

merging?
		Linus

--

Well, no, but actually there are only a few of my patches in this merge

window. :-)
Moreover, if the maintainers who took them told me they would be scheduled for

the next merge window, I wouldn't mind.  That actually happended to some of my

patches that are in the Greg's tree at the moment and that's fine (although I

consider the patches as important).
IMO, this is a question of balance.  Of course, a maintainer can take

everything from everyone, but at the same time he can have a look at the

patches and say "Well, I have lots of stuff scheduled for this merge window

already, this stuff of yours will wait for the next merge window.  Please

improve the code or review the others' patches in the meantime".  The only
I think the majority of developers would understand if you told them you could

only merge a limited amount of changes in a single merge window, provided that

they would be treated fairly.
When you take everything from everyone, you actually reward people who are

able to develop more code between merge windows.  Not necessarily those who

spend time on different important activities, such as reviewing the others'
That shouldn't be necessary. :-)
The point is to tell people to develop the code less rapidly, so to speak.

Or maybe more carefully.
Thanks,

Rafael

--

I think you're approaching a solution Linus. If developers take a refusal

as a punishment, maybe you can use that for trees which have too many

unresolved regressions. This would be really unfair to subsystem maintainers

which themselves merge a lot of work, but recursively they may apply the

same principle to their own developers, so that everybody knows that it's

not worth working on next code past a point where too many regressions are

reported.
Willy
--

Heh. It's been done. In fact, it's done all the time on a smaller scale.

It's how I've enforced some cleanliness or process issues ("I won't pull

that because it's too ugly"). I see similar messages floating around about

individual patches.
That said, I don't think it really works that well as "the solution": it

works as a small part of the bigger picture, but no, we can't see

punishment as the primary model for encouraging better bevaiour. 
First off, and maybe this is not true, but I don't think it is a very

healthy way to handle issues in general. I may come off as an opinionated

bastard in discussions like these, and I am, but when it actually comes to

maintaining code, really prefer a much softer approach.
I want to _trust_ people, and I really don't want to be a "you need to do

'xyz' or else" kind of guy. 
So I'll happily say "I can't merge this, because xyz", where 'xyz' is

something that is related to the particular code that is actually merged.

But quite frankly, holding up _unrelated_ fixes, because some other issue

hasn't been resolved, I really try to not do that. 
So I'll say "I don't want to merge this, because quite frankly, we've had

enough code for this merge window already, it can wait". That tends to

happen at the end of the merge window, but it's not a threat, it's just me

being tired of the worries of inevitable new issues at the end of the

window.
And I personally feel that this is important to keep people motivated.

Being too stick-oriented isn't healthy.
The other reason I don't believe in the "won't merge until you do 'xyz'"

kind of thing as a main development model is that it traditionally hasn't

worked.  People simply disagree, the vendors will take the code that their

customers need, the users will get the tree that works for them, and

saying "I won't merge it" won't help anybody if it's actually useful.
Finally, the people I work with may not be perfect, but most maintainers

are pretty much experts within their ...
[ message continues ]

And, ideally, they would have posted the changes as patches to the list

for review anyway, so there shouldn't be anything surprising in that pull...
    J

--

yes, it's something which has been disappearing since use of bk then git.

It would be impratical and useless to post everything during the merge

window now, but if we can get everyone to pass through linux-next, the

posts will be evenly distributed and it would make sense to require

everyone to post their changes to the list at the same time. Right now,

some developers already always post their changes. Jeff, Greg and

Bartlomiej come to mind, and I must say that I'm always interested in

performing a quick look, just in case something really obvious catches

my attention (which never happens).
Willy
--

It may help directly, for example when people realize that they work on
I totally agree with that.
Still, the issue at hand is that

(1) The code merged during a merge window is somewhat opaque from the tester's

     point of view and if a regression is found, the only practical means to

    figure out what caused it is to carry out a bisection (which generally is

    unpleasant, to put it lightly).

(2) Many regressions are introduced during merge windows (relative to the

    total amount of code merged they are a few, but the raw numbers are

    significant) and because of (1) the process of removing them is generally

    painful for the affected people.

(3) The suspicion is that the number of regressions introduced during merge

    windows has something to do with the quality of code being below

    expectations, that in turn may be related to the fact that it's being

    developed very rapidly.
My opinion is that we need to solve this issue sooner rather than later and so

the question is how we are going to approach that.
Thanks,

Rafael

--

Hey, guv, do you _honestly_ believe that some kind of ISO-9000-like

process generates quality?
And I dislike how people try to conflate "quality" and "merging speed" as

if there was any reason what-so-ever to believe that they are related.
You (and Andrew) have tried to argue that slowing things down results in

better quality, and I simply don't for a moment believe that. I believe

the exact opposite.
The way to get good quality is not to put barriers up in front of

developers, but totally the reverse - by helping them. And yes, that help

can quite possibly be in the form of "process" - by making things more

streamlined, and by having people not have to waste time on wondering

where they should send things etc.
But the notion that we should even _try_ to aim to slow things down, that

one I find unlikely to be true, and I don't even understand why anybody

would find it a logical goal?
Of course, you will have fewer new bugs if you have fewer changes. But

that's not a goal, that's a tautology and totally uninteresting. A small

program is likely to have fewer bugs, but that doesn't make something

small "better" than something large that does more.
Similarly, a stagnant development community will introduce new bugs more

seldom. But does that make a stagnant one better than a virbrant one? Hell

no.
So what I'm arguing against here is not that we should aim for worse

quality, but I'm arguing against the false dichotomy of believing that

quality is incompatible with lots of change. 
So if we can get the discussion *away* from the "let's slow things down",

then I'm interested. Because at that point we don't have to fight made-up

arguments about something irrelevant.
			Linus

--

Note that I'm not necessarily arguing for slowing down, but for reduced

functional conflicts (which slow down may help but it's not the only

solution). I think that refining the time resolution might achieve the

same goal. Instead of merging 10000 changes which each have 1% chance

of breaking any other area, and have all developers try to hunt bugs

caused by unrelated changes, I think we could do that in steps.
To illustrate, instead of changing 100 areas with one of them causing

breaking in the other ones, and having 100 victims try to hunt the

bug in 99 other areas, then theirs, and finally insult the faulty

author, we could merge 50 areas in version X and 50 in X+1 (or 3*33

or 4*25, etc...). That way, we would only have 50 victims trying to

find the bug in 49 other areas (or 32 or 24). Less people wasting

their time will mean faster validation of changes, and possibly

faster release cycle with better quality.
People send you their crap every two months. If you accept half of

it every month, they don't have to sleep on their code, and at the

same time at most half of them are in trouble during half the time
Willy
--

How about:
(1) Merge a couple of trees at a time (one tree at a time would be ideal, but

    that's impossible due to the total number of trees).

(2) After (1) give testers some time to report problems introduced by the

    merge.

(3) Wait until the most urgent problems are resolved.  Revert the offending

    changes if there's no solution within given time.

(4) Repeat for another couple of trees.

(5) Arrange things so that every tree gets merged once every two months.
This would also give us an idea of which trees introduce more problems.
Thanks,

Rafael

--

You can't get there from here (at least not very easily).
If you have 60 trees, and want a merge for each one every 2 months, you have to

average 1 tree a day.  How big a delay you want in step (2) directly impacts

how many trees you merge at once - if you want a week of cook time, you have to

merge 7 trees every Monday, and so on...

...and what would you do with such information?
I'm not actually worried about my tree but if (theoretically) it happens to

be amongst the "problematic" ones I would be a bit pissed by blame shifting,

especially given that it is very difficult to compare different trees as

they (usually) deal with quite different areas of the code (some are messy

and problematic, yet critical while others can be more forgiving).
Also slowing down things to focus on quality is really a bad idea.  You can

trust me on this one, I've tried it once on the smaller scale and it was a

big disaster cause people won't focus on quality just because you want them

to.  They'll continue to operate in the usual way and try to workaround you

instead (which in turn causes extra tensions which may become quiet warfare).

In the end you will have a lot more problems to deal with...
Same goes for any other kind of improvement by incorporating "punishment" as

the part of the process.  You are much better helping people and trying them

to understand that they should apply some changes to their way of work because

it would be also beneficial for _them_, not only for _you_.
Now regarding the development model - I think that there is really no need

for a revolution yet, instead we should focus on refining the current process

(which works great IMO), just to summarize various ideas given by people:
- try to persuade few black sheeps that skipping linux-next completely for

  whole patch series is a really bad idea and that they should try to spend

  a bit more time on planning for merge instead of LastMinute assembly+push

  (by doing it right they could spend more time after merge to prepare for

  the next one or fixing old bugs instead of chasing new regressions, overall

  they should have _more_ time for development by doing it right)
- encourage flatting of merges during the merge window so instead of 1-2 big

  merges per tree at the beginning of the merge you have few smaller ones

  (majority of maintainers do it...
[ message continues ]

On Thu, May 1, 2008 at 6:16 AM, Bartlomiej Zolnierkiewicz
When a teacher assigns grades in a class, it's not punishment, it's feedback.
I don't think anyone *intends* to push crap into the tree. However,

with the barrier to getting things into the tree so low, some may feel

there's less incentive to try to get things right the first (or

second) time. It would be nice to provide that incentive.
Normally, it'd be peer-review of the uncommitted patches. We don't

have a lot of that going on here, though. So,

peer-review-after-the-fact, ie, who placed this massive turd in the

tree, and everyone swivels an eye over there and asks what went wrong,

and how do we prevent it in the future. Those conversations seem to be

happening already, time to time.
And as a policy suggestion, if we're past rc1 and someone has

identified a commit as the root of a regression/bug, then the policy

should be just to revert it immediately, no questions asked. Let the

original author work with the person who identified the problem and

resend a fixed commit later. We lose testers in the meantime, and

perhaps the extra effort involved in having the author work out the

issues and redo the patch will help prevent drive-by patching in the

future.

--

you make a valid point here : "we lose testers in the meantime". Maybe

it would help if -rc2 would be released a few days after -rc1 with

the first most obvious showstoppers (often build issues). The most

problematic ones are often fixed within an hour or so, but for most

testers, it still means they have to wait for -rc2.
Most external testers might then only try -rc2 first, but that's not

a problem. What we really want is them to test widely and not revert

back at the first problem. If only 20% of testers try -rc1, and the

remaining 80% actively wait for -rc2 3 days after, then we'll get

broader testing in the first two weeks.
Willy
--

There still are too many bugs of this kind that make it to the Linus' tree and
Well, I'm not sure what that's supposed to mean, so I won't comment.
Thanks,

Rafael

--

Agreed but if you trace the way of these bugs into the Linus' tree many of

them follow one of two patterns:
* -mm / -next skipped completely
* short time in -mm / -next (< 2 weeks)
[ disclaimer: this is based on my observations, no hard data to prove it ]
Please also remember that linux-next concept is still quite _fresh_ with

a _plenty_ of room for enhancements like having kernel-du-jour packages for

the most popular distros, doing more automated testing + searching for
This was not directed at you (you are doing great work BTW) but rather

at some people trolling the thread.
Thanks,

Bart

--

Perhaps it would make sense to split the merge window into 2 - first

week kernel/net/mm/lib etc., second week arch/drivers/fs? Obviously

some changes are going to span those two areas but it might help in

pinpointing where breakage was introduced as well as quietening the

thundering herd of pull requests at the start of a merge window and

thereby allow review to happen over a longer period.
Or I could just be dreaming...

--

Sorry, not Andrew. DavidN.
Andrew argued the other way (quality->slower), which I also happen to not

necessarily believe in, but that's a separate argument.
Nobody should ever argue against raising quality.
The question could be about "at what cost"? (although I think that's not

necessarily a good argument, since I personally suspect that good quality

code comes from _lowering_ costs, not raising them).
But what's really relevant is "how?"
Now, we do know that open-source code tends to be higher quality (along a

number of metrics) than closed source code, and my argument is that it's

not because of bike-shedding (aka code review), but simply because the

code is out there and available and visible.
And as a result of that, my personal belief is that the best way to raise

quality of code is to distribute it. Yes, as patches for discussion, but

even more so as a part of a cohesive whole - as _merged_ patches!
The thing is, the quality of individual patches isn't what matters! What

matters is the quality of the end result. And people are going to be a lot

more involved in looking at, testing, and working with code that is

merged, rather than code that isn't.
So _my_ answer to the "how do we raise quality" is actually the exact

reverse of what you guys seem to be arguing.
IOW, I argue that the high speed of merging very much is a big part of

what gives us quality in the end. It may result in bugs along the way, but

it also results in fixes, and lots of people looking at the result (and

looking at it in *context*, not just as a patch flying around).
And yes, maybe that sounds counter-intuitive. But hey, people thought open

source was counter-intuitive. I spent years explaining why it should work

at all!
			Linus

--

And we introduce bugs that nobody sees until they appear in a CERT advisory.
IMnsHO, the quick merging results in lots of code that nobody looked at,

except for the author, nobody is looking at and nobody will _ever_ look at.

Simply, because there's no time for looking at that code, since we're

supposed to be working on preparing new code for the next merge window, testing

the already merged code etc., around the clock.  Now, you may hope that this

not-looked-at-by-anyone code is of high quality nevertheless, but I somehow

doubt it.
[Note that it's not directly related to the issue at hand, which is the fact

that people affected by regressions are heavily punished by our current

process.  Never mind, though.]
And that's not to mention bugs that appear in the code everybody looked at

and happily reach the mainline because that code has not been tested well

enough before merging.  Take SLUB as an example, if you wish.
The fact is, we're merging stuff with minimal-to-no review and with minimal

testing reasonably possible.  Is _that_ supposed to produce the high quality?
Also, I'm not buying the argument that the quality of code improves over time

just because it's open and available to everyone.  That only happens to the

code which is actually looked at by someone or attempted to modify.  This

obviously doesn't apply to the whole kernel code.
For this reason, IMO, we should do our best to ensure that the code being

merged is of high quality already at the moment we merge it.  How to achieve

that is a separate issue.
BTW, we seem to underestimate testing in this discussion.  In fact, the vast

majority of kernel bugs are discovered by testing, so perhaps the way to go

is to make regular testing of the new code a part of the process.
Thanks,

Rafael

--

On Thu, 1 May 2008 13:38:33 +0200
well.. -rc1 to -rc8 are doing that already, somewhat.

Can we do better? Always. The more testing the better, and the more

testers the better.

--

The testing is not really a part of the process right now, though.  We somehow

hope that the kernel will be tested sufficiently before a major release, but

we don't measure the testing coverage, for example.  Of course, that will

involve more work independent of the code writing, but at one point it'll

just become a necessity.
Thanks,

Rafael

--

On Thu, 1 May 2008 14:41:05 +0200
Well. Take 2.6.25.. we know Fedora shipped it in their alpha's and

betas (and in rawhide). Those are used by a lot of people; so for me

that's a whole bunch of coverage right there. Is it perfect? No.

But in a way it's in the spirit of open source: the people who care

about a stable release the most (distros) [1], helped us getting this

tested. The other people on this thread we care greatly at least also

help us test in general.
[1] Not trying to say no single person wouldn't care; but a distro

tends to care more due to the sheer number of users...

--

The main problem as I see it is with the huge number of hard, confirmed bugs

that are *not* getting fixed.
With the current development model, developers only really care about

current regressions. In a large part this is due to the excellent work of

Rafael with his tracking of regressions since the previous release.

But it does mean older regressions fall by the wayside, even if they've been

confirmed, bisected and the submitter is responsive.

For a while Natalie Protasevich did some work on trying to get attention for

older regressions, but that effort seems to have died out.
Two concrete examples from my personal experience:

- http://bugzilla.kernel.org/show_bug.cgi?id=9749; the error:

  sysctl table check failed:

  /dev/parport/parport0/devices/ppdev0/timeslice  Sysctl already exists

  First reported for 2.6.24-rc5, just now confirmed with 2.6.25

  Acknowledged by maintainer, but no follow-up [1].
- http://bugzilla.kernel.org/show_bug.cgi?id=9310; the error:

  completely blank console with FRAMEBUFFER_CONSOLE_DETECT_PRIMARY set when

  framebuffer is active, but no VGA=xxx parameter is passed

  First reported for 2.6.23, confirmed for 2.6.24-rc6, almost certainly

  still present in 2.6.25

  Acknowledged by maintainer, but no follow-up despite later pings.
Another issue is that sometimes developers really are too eager to get their

changes into mainline even when there are known issues or when they know in

their heart that the changes have not received enough testing.
Example is a a scheduler change [2] that causes a completely reproducible

regression (music skips and key repeats) on my box with one specific

workload. Ingo and Peter have been great doing debugging after I reported

it for 2.6.25-rc8 and it was reverted just before the release, but I was

very surprised to see the patch resubmitted for 2.6.26 without the

regression being resolved first.

It is now confirmed to still be there and there has been additional effort

on it, but so far without resu...
[ message continues ]

Hi.
No. People generally expect that code that has been merged does work, so

they don't look at it unless they're forced to (by a bug or the desire

to make further modifications in that code) and they don't explicitly

seek to test it. They just seek to use it.
When it doesn't work, some of us will go and seek to find the cause,

others (most?) will simply roll back to whatever they last found to be

reliable.
Out of tree code has the same issues.
The only time code really gets looked at and tested is when there's a

problem, or when people are explicitly choosing to inspect it (pre-merge

reviews, eg).
So my answer to the "how do we raise quality" question would be that

when writing the code, we put time and effort into properly analysing

the problem and developing a solution, we put time and effort into

carefully testing the solution, and we put code in that will help the

end-user help us to debug issues later (without them necessarily needing

to git-bisect). After all, good software isn't the result of random (or

semi-random), unconsidered modifications, but of planning, thought and

attention to detail.
In other words, I'm arguing that the speed of merging should be

irrelevant. What's relevant is the quality of the work done in the first

place.
If you want better quality code, penalise the people who get buggy code

merged. Give them a reason to get it in a better state before they try

to merge. Of course Linus alone can't do that.
Nigel
--

Amen!
-- 
"They that give up essential liberty to obtain temporary safety,

deserve neither liberty nor safety."  (Ben Franklin)
"The course of history shows that as a government grows, liberty

decreases."  (Thomas Jefferson)
--

Really?  And how, pray tell, being out there will magically improve the

code?  "With enough eyes all bugs are shallow" stuff out of ESR's arse?
FWIW, after the last month's flamefests I decided to actually do something

about review density of code in the areas I'm theoretically responsible

for.  Namely, do systematic review of core data structure handling (starting

with the place where most of the codepaths get into VFS - descriptor tables

and struct file), doing both blow-by-blow writeup on how that sort of things

is done and documentation of the life cycle/locking rules/assertions made

by code/etc.  I made one bad mistake that held the things back for quite

a while - sending heads-up for one of the worse bugs found in process to

never-sufficiently-damned vendor-sec.  The last time I'm doing that, TYVM...
Anyway, I'm going to get the notes on that stuff in order and put them in

the open.  I really hope that other folks will join the fun afterwards.

The goal is to get a coherent braindump that would be sufficient for

people new to the area wanting to understand and review VFS-related code -

both in the tree and in new patches.
files_struct/fdtable handling is mostly dealt with, struct file is only

partially done - unfortunately, struct file_lock has to be dealt with

before that and it's a (predictable) nightmare.  On the other end of

things, fs_struct is not really started, vfsmount review is partially

done, dentry/superblock/inode not even touched.
Even with what little had been covered... well, let's just say that it

caught quite a few fun turds.  With typical age around 3-4 years.  And

VFS is not the messiest part of the tree...

--

In the same way that ESR's arse would improve if he'd not wear pants: by him

going to the gym more to avoid at least a few of the many disgusted stares.
ie, the magic would be in the quality of the code being greater simply due

to developer being aware of the openness. The effect probably wears of after

enough time though...
Rene.

--

thank you, the lack of good documentation on the intent of the code has

been a significant barrier for new people. it's (relativly) easy for a

good programmer to look at the code and figure out how it does things, a

bit harder to figure out what it does, but why it does it (and what it was 
it may not be the messiest part of the tree, but it's definantly one of

the hardest to figure out the intent of.
David Lang

--

From: Linus Torvalds <torvalds@linux-foundation.org>
This is a huge burdon to put on people.
The more broken stuff you merge, the more people are forced to track

these problems down so that they can get their own work done.
It punishes people who do put forth the effort to let new changes cook

properly, before pushing, and thus avoid putting turds into the tree.
You really have to think about the ramifications of this system.

--

I'm not saying we should merge crap.
You can take any argument too far, and clearly it doesn't mean that we

should just accept *anything*, because it will magically be gilded by its

mere inclusion into the kernel. No, I'm not going to argue that.
But I do want to argue against the notion that the only way to raise

quality is to do it before it gets merged. It's often better to merge

early, and fix the issues the merge brings up early too!
Release early, release often. That was the watch-word early in Linux

kernel development, and there was a reason for it. And it _worked_. Did it

mean "release crap, release anything"? No. But it did mean that things got

lots more exposure - even if those "things" were sometimes bugs.
			Linus

--

From: Linus Torvalds <torvalds@linux-foundation.org>
That's exactly what's been happening this merge window though.
And throughout this, Andrew Morton has been the only person with the

balls and lack of ego problems to revert regression causing changes he

introduced.

--

eh?  I argued the opposite: that increasing quality will as a side-effect

slow things down.
If we simply throttled things, people would spend more time watching the

shopping channel while merging smaller amounts of the same old crap.
--

Yes, my bad, I realized that when I read through my message and already 
I agree totally. And although some of the time would probably _also_ be

spent on the frustrating crap that was designed to do the throttling, that

isn't much more productive than watching the shopping channel would be ...
		Linus

--

Okay, so what exactly are we going to do to address the issue that I described

in the part of my last message that you skipped?
Rafael

--

And quite frankly, (2) and (3) are both: "merge windows introduce new

bugs", and that's such an uninteresting tautology that I'm left

wordless.  And (1) is just a result of merrging lots of stuff.
Of course the new bugs / regressions are introduced during the merge

window.  That's when we merge new code.  New bugs don't generally happen

when you don't get new code. 
And of course finding bugs is always painful to everybody involved.
And of course the bugs indicate something about the quality of code

being merged.  Perfect code wouldn't have bugs.
So what you are stating isn't interesting, and isn't even worthy of

discussion.  The way you state it, the only answer is: don't take new

code, then.  That's what your whole argument always seems to boild down

to, and excuse me for (yet again) finding that argument totally

pointless. 
So let me repeat:
 (1) we have new code. We always *will* have new code, hopefully. A few

     million lines pe year.
     If you don't accept this, I don't have anything to say.
 (2) we need a merge window.  That is a direct result not of wanting to

     have lots of code at the same time, but of the _reverse_ issue: we

     want to have times of relative calm.
     And again, if you continue to see the merge window as the

     "problem", rather than as the INEVITABLE result of wanting to have

     a calm period, there's no point in talking to you. 
 (3) Ergo, there's a very fundamental and basic and inescapable result:

     we absolutely _will_ have times when we get lots and lots of new

     code. 
So these are not "problems".  They are *facts*.  Stating them as

problems is stupid and pointless.  I'm not going to discuss this with

you if you cannot get over this. 
So please accept the facts.
Once you accept the facts, you can state the things you can change.  But

the things you cannot change is the merge window, and the fact that we

get a lot of new code at a high rate (where the merge window will

inevitably compress tha...
[ message continues ]

Pardon this comment from an inexperienced kernel hacker, but it seems to

me that one of the main problems is subsystems stomping on each other

during the merge window, and a general confusion as to who is responsible

for what bugs that appear.
Perhaps a shorter merge window, using a round-robin approach, based on

subsystem, would help alleviate these issues?
This would:
	- give people a "known" tree to base their subsystem patches on,

		when their turn comes around
	- give a rough schedule if the round-robin was always consistent

		in order, or made known in advance
	- a shorter window would keep people from waiting too long for

		their turn
	- give those responsible for the currently merged subsystem

		motivation and clarity to fix bugs that do appear during

		their merge window
Problems I see with this approach:
	- those at the end of the cycle get the shaft, if previous changes

		affect their work
	- political issues with determining the order of the round-robin

		schedule
If I'm overlooking something, I'm sure someone will correct me. :-)
- Chris
--

Hi folks,
<big_snip>
Just a few naive thoughts:
a) What about reducing code size ?
Some parts, IMHO, doen't necessarily need to be in the kernel,

eg. certain filesystems. Less code, less patches to review, less

chance of kernel bugs. Of course this might also cause other

impacts (eg. performance), so those decisions require great care.
b) Mutli-tier trees / patchlines
IMHO, a major problem are conflicting patches (eg. a core change

causes some driver to break). In measurement instrumentation

(eg. timesync), there's typically one primary reference point

(eg. atomic clock) as tier-0, where (a limited set of) tier-1's

are synchronized against, tier-2 syncs against tier-1 and so on.
So for the linux kernel, we perhaps could have something like:
* tier-0: core

* tier-1: arch

* tier-2: hw drivers

* tier-3: sw drivers

* tier-4: userland interfaces
If a change from a lower tier wants to it's upper tier, it first

MUST fit it's current mainline and carefully checked. Of course

this introduces longer times for an individual change to go to

into release (since it has to pass several tiers), but IMHO the

chance of new bugs in release should be reduced this way.
Of course there might be chances in a lower tier, which obviously

won't affect several intermediate tiers. Those could skip some tiers.
For example, I'm currently working on an /proc interface for

changing process privileges. In my model, this had to be settled

in #4, but shouldn't touch drivers (#2,#3), but maybe arch (#1).

So these changes could be kicked directly to #2. 
What do you think about this ?
cu

--

---------------------------------------------------------------------

 Enrico Weigelt    ==   metux IT service - http://www.metux.de/

---------------------------------------------------------------------

 Please visit the OpenSource QM Taskforce:

 	http://wiki.metux.de/public/OpenSource_QM_Taskforce

 Patches / Fixes for a lot dozens of packages in dozens of versions:

	[ message continues ]