From: James Bottomley <James.Bottomley@...>
Subject: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 1:20 pm 2008

In the spirit of having a more process than technical based kernel
summit, I'd like to put the topic of the kernel Janitors project up for
discussion.

In the early days, the project was conceived as a way of getting fresh
blood into kernel development by giving them fairly simple but generally
useful tasks and hoping they'd move more into the mainstream.  If we
wind forwards to 2008, there's considerable and rising friction being
generated by janitorial patches.  This is only an example:

http://marc.info/?l=linux-kernel&m=121135889328760

but there are many more.  The greatest problem, as I see it is that by
pouring vitriol like this on newbies, we're really damaging our
reputation as a community that welcomes newcomers and strangling our
necessary supply of willing volunteers.  On the other hand, as a
maintainer, when there's people yelling me at about patches not being
included plus a persistent regressions list and about ten bug reports to
track down, the last thing I want to see within a million miles of my
inbox is a white space fixing patch.  The more of these patches we get,
the worse the problem becomes and the shorter and more inflammatory the
responses get. We can't go on like this.

The most obvious solution might be to shut the Janitors project down, or
at least more tightly manage its TODO list (although a lot of what gets
seen as janitorial patches, like whitespace fixes, isn't on the TODO
list in the first place).  However, since the purpose is to get new
people involved with kernel development, perhaps we should repurpose the
project so it actually does this.  My suggestion is that we replace it
with the kernel bugs project.  Kudos for finding bugs, more for finding
better ways of finding bugs, and the most for finding and actually
fixing a bug.  

Perhaps we should simply start the discussion with the premise that we
want to encourage new people to do useful work and draw them into the
development community and see where it leads.

James



--


From: Rik van Riel <riel@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 3:08 pm 2008

On Wed, 28 May 2008 12:20:12 -0500
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> The most obvious solution might be to shut the Janitors project down, or
> at least more tightly manage its TODO list 

I've had some limited success with:

http://kernelnewbies.org/KernelProjects

One problem is that people end up beginning with a project but
for some reason stop working on it later - with no indication
as to whether the project ended up being too difficult, or they
ran out of time, or something else happened...

-- 
All rights reversed.
--


From: Chris Mason <chris.mason@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 4:15 pm 2008

On Wednesday 28 May 2008, Rik van Riel wrote:
> On Wed, 28 May 2008 12:20:12 -0500
>
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > The most obvious solution might be to shut the Janitors project down, or
> > at least more tightly manage its TODO list
>
> I've had some limited success with:
>
> http://kernelnewbies.org/KernelProjects
>
> One problem is that people end up beginning with a project but
> for some reason stop working on it later - with no indication
> as to whether the project ended up being too difficult, or they
> ran out of time, or something else happened...

I think our mistake is the assumption that everyone who wants to contribute to 
the kernel wants to code, or that everyone wants to end up at the top of the 
major feature contributors list.  For lots of people, we're going to be that 
experiment from college they never quite want to admit to later on in life.

Looking at the KernelProjects link, bugzilla is at the buttom and janitors is 
at the top.  I've always thought the janitors project was important, but 
maybe we want to change the emphasis around a bit.

The regressions page on kernelnewbies is out of date, and there are many more 
howtos on coding than on bug hunting.  It doesn't mention linux-next 
anywhere.

I'm not picking on kernelnewbies, it is one of our best resources.  But, going 
back to James' ideas, introducing people to bug hunting is a better way to 
involve them in the community.

Maintainers are easier to interact with when you send good bug reports along 
with a clear way to reproduce it and perhaps a bisection

Getting all of the above is often 95% of the work of fixing it, people are 
most likely to jump into the coding when they've found a bad patch and start 
to wonder if they can fix it themselves.

Along those lines, how about a kernel bug hunting challenge.  The top 
contributor(s) to creating bugzillas that get solved get a new pc, laptop, 
high end graphics card, ssd device...whatever.  We could send the best 
testers hardware that breaks most often...everyone wins.

Other prizes could include tickets to a linux conf of choice (not airline 
tickets, just free entry).

One motivation here is to bring bug reporters from active distro communities 
into testing mainline kernels as well.

-chris
--

From: David Woodhouse <dwmw2@...>
Subject: Re: [Ksummit-2008-discuss]  Fixing the Kernel Janitors project
Date: May 28, 4:49 pm 2008

On Wed, 2008-05-28 at 12:20 -0500, James Bottomley wrote:
> In the early days, the project was conceived as a way of getting fresh
> blood into kernel development by giving them fairly simple but generally
> useful tasks and hoping they'd move more into the mainstream.  If we
> wind forwards to 2008, there's considerable and rising friction being
> generated by janitorial patches.  This is only an example:
> 
> http://marc.info/?l=linux-kernel&m=121135889328760
> 
> but there are many more.

The example that sums it up for me is this one:

	http://lkml.org/lkml/2008/5/18/20

We have people making minor cosmetic changes, and not paying even the
_slightest_ attention to what they're doing. This one's a particularly
scary example because it's something even the most non-technical person
should have spotted; there's _no_ excuse. It's the cosmetic equivalent
of a naïve warning fix that leaves the actual bug in place.

I think you're right that the status quo is damaging, and I don't see it
getting any better with the current quality of 'janitoring'. I think the
only way we can salvage anything useful from the janitors project is to
keep a close rein on what tasks are actually undertaken.

But we've pushed back on people doing this kind of thing before, and
pointed them both at the obvious things they've missed in the context of
their patches, and other more useful things they could be doing -- but
we've often received responses along the lines of "but I don't want to
have to _think_!".

It's hard to know where to go from there, and it's not exactly
surprising that we end up frustrated.

-- 
dwmw2

--


From: James Bottomley <James.Bottomley@...>
Subject: Re: [Ksummit-2008-discuss]  Fixing the Kernel Janitors project
Date: May 28, 5:01 pm 2008

On Wed, 2008-05-28 at 23:49 +0300, David Woodhouse wrote:
> On Wed, 2008-05-28 at 12:20 -0500, James Bottomley wrote:
> > In the early days, the project was conceived as a way of getting fresh
> > blood into kernel development by giving them fairly simple but generally
> > useful tasks and hoping they'd move more into the mainstream.  If we
> > wind forwards to 2008, there's considerable and rising friction being
> > generated by janitorial patches.  This is only an example:
> > 
> > http://marc.info/?l=linux-kernel&m=121135889328760
> > 
> > but there are many more.
> 
> The example that sums it up for me is this one:
> 
> 	http://lkml.org/lkml/2008/5/18/20
> 
> We have people making minor cosmetic changes, and not paying even the
> _slightest_ attention to what they're doing. This one's a particularly
> scary example because it's something even the most non-technical person
> should have spotted; there's _no_ excuse. It's the cosmetic equivalent
> of a naïve warning fix that leaves the actual bug in place.
> 
> I think you're right that the status quo is damaging, and I don't see it
> getting any better with the current quality of 'janitoring'. I think the
> only way we can salvage anything useful from the janitors project is to
> keep a close rein on what tasks are actually undertaken.
> 
> But we've pushed back on people doing this kind of thing before, and
> pointed them both at the obvious things they've missed in the context of
> their patches, and other more useful things they could be doing -- but
> we've often received responses along the lines of "but I don't want to
> have to _think_!".
> 
> It's hard to know where to go from there, and it's not exactly
> surprising that we end up frustrated.

Right, but that's why I think we have to change the process.  If we keep
the Janitors project, then the bar has to be raised so that it becomes
more participatory and thought oriented (i.e. eliminate from the outset
anyone who is not going to graduate from mechanical changes to more
useful ones).  That's why I think bug finding and reporting is a better
thing to do.  There are mechanical aspects to finding bugs but it is a
useful service.  Bug fixing is participatory because we usually do quite
a lot of back and forth between the reporter and the fixer and at the
end of the day quite a few people get curious about how the bug was
triggered in the first place and actually go off and read the code.

James


--


From: Pekka Enberg <penberg@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 5:31 pm 2008

Hi James,

On Thu, May 29, 2008 at 12:01 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> Right, but that's why I think we have to change the process.  If we keep
> the Janitors project, then the bar has to be raised so that it becomes
> more participatory and thought oriented (i.e. eliminate from the outset
> anyone who is not going to graduate from mechanical changes to more
> useful ones).

I'm not sure what you expect to happen if we "shut down" the Janitors
project. The important janitorial work doesn't just magically
disappear. For example, we still need people for:

  - Fixing API misuse
  - Converting code from old APIs to new ones
  - Consolidating duplicate code
  - Fixing error handling code
  - Removing unused code
  - De-obfuscating code (e.g. removing bad macro magic, etc.)

And, quite frankly, I don't see what the big fuss is about. I know
we've had way too many "whitespace cleanup" patches in the past six
months or so, but can't we just NAK them politely and be done with it?

                        Pekka
--


From: James Bottomley <James.Bottomley@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 5:42 pm 2008

On Thu, 2008-05-29 at 00:31 +0300, Pekka Enberg wrote:
> On Thu, May 29, 2008 at 12:01 AM, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
> > Right, but that's why I think we have to change the process.  If we keep
> > the Janitors project, then the bar has to be raised so that it becomes
> > more participatory and thought oriented (i.e. eliminate from the outset
> > anyone who is not going to graduate from mechanical changes to more
> > useful ones).
> 
> I'm not sure what you expect to happen if we "shut down" the Janitors
> project. The important janitorial work doesn't just magically
> disappear. For example, we still need people for:

I'm just outlining the possible solutions; shutting it down wasn't the
one I advocated. One can argue that janitorial changes with enough
intrinsic value tend to get done anyway regardless of whether we have
the project or not.

>   - Fixing API misuse
>   - Converting code from old APIs to new ones
>   - Consolidating duplicate code
>   - Fixing error handling code
>   - Removing unused code
>   - De-obfuscating code (e.g. removing bad macro magic, etc.)
> 
> And, quite frankly, I don't see what the big fuss is about. I know
> we've had way too many "whitespace cleanup" patches in the past six
> months or so, but can't we just NAK them politely and be done with it?

The problem is that there's something in the way all of this is working
that's causing politeness to get short shrift. In turn that's giving
lkml a larger than normal reputation for being a free for all dog fight
and discouraging potential contributors from coming forwards.

Appeals to be politer tend to only work in the short term (having given
quite a few of them).  I think we're developing a root cause problem in
the way we recruit people to work in the kernel and we have to think
about fixing it there.

James


--


From: David Woodhouse <dwmw2@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 6:18 pm 2008

On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> Appeals to be politer tend to only work in the short term (having given
> quite a few of them).  I think we're developing a root cause problem in
> the way we recruit people to work in the kernel and we have to think
> about fixing it there.

Do we really have a problem recruiting people to work in the kernel? On
what do you base that observation?

On a similar note, do we have any real data on the question of whether
those who are volunteering the patches which raise so much ire would
_ever_ become productive members of the team, even if we were to nurture
them properly?

-- 
dwmw2

--


From: James Bottomley <James.Bottomley@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 6:35 pm 2008

On Thu, 2008-05-29 at 01:18 +0300, David Woodhouse wrote:
> On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> > Appeals to be politer tend to only work in the short term (having given
> > quite a few of them).  I think we're developing a root cause problem in
> > the way we recruit people to work in the kernel and we have to think
> > about fixing it there.
> 
> Do we really have a problem recruiting people to work in the kernel? On
> what do you base that observation?

Yes, the median age of the MAINTAINERS is rising.  Not quite at the rate
of a year per year which would show we have practically no turn over,
but it is rising.

However, even if there were no recruitment problem at all, getting more
people involved is always better because it means more contributions.
And contributions (useful ones) are the lifeblood that moves the kernel
forwards.

> On a similar note, do we have any real data on the question of whether
> those who are volunteering the patches which raise so much ire would
> _ever_ become productive members of the team, even if we were to nurture
> them properly?

I don't think so.  But that's not really what I'm saying.  I'm saying we
need to make the process of encouraging useful contributors more
streamlined (as in less aggro on the mailing list).  If that involves
cutting out the less useful ones earlier, so be it.  If we can come up
with a better conversion process, that's great too.  I want to start the
discussion, not necessarily prescribe the solution.

James


--


From: David Miller <davem@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 29, 2:09 am 2008

From: James Bottomley <James.Bottomley@HansenPartnership.com>
Date: Wed, 28 May 2008 17:35:27 -0500

> However, even if there were no recruitment problem at all, getting more
> people involved is always better because it means more contributions.

This is not true at all.

If people are getting involved, just for the sake of being involved,
which there is strong evidence of, then it's not a positive thing.

We want people who are passionate about doing things with the
kernel, are self-motivated, and frankly don't need a ton of hand
holding and do not work on things that require absolutely no
thinking.

Look at anyone who is extremely nimble with the kernel, and ask them
what they worked on to get going with development.  Did Andrew Morton
fixup whitespace errors when he was starting to become familiar with
the tree?  Did I?  No, none of us did this stuff.  We read over the
code and learned how it worked, did a port, optimized a lookup
algorithm somewhere.

Consistently we see people turding with whitespace, and not breaking
out of that cycle.  That is a problem.
--


From: Greg KH <greg@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 6:51 pm 2008

On Wed, May 28, 2008 at 05:35:27PM -0500, James Bottomley wrote:
> On Thu, 2008-05-29 at 01:18 +0300, David Woodhouse wrote:
> > On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> > > Appeals to be politer tend to only work in the short term (having given
> > > quite a few of them).  I think we're developing a root cause problem in
> > > the way we recruit people to work in the kernel and we have to think
> > > about fixing it there.
> > 
> > Do we really have a problem recruiting people to work in the kernel? On
> > what do you base that observation?
> 
> Yes, the median age of the MAINTAINERS is rising.  Not quite at the rate
> of a year per year which would show we have practically no turn over,
> but it is rising.

My raw numbers show that the number of individual kernel contributors
continues to increase with every release, so this might not be as much
of a problem as it's made out to be.

> However, even if there were no recruitment problem at all, getting more
> people involved is always better because it means more contributions.
> And contributions (useful ones) are the lifeblood that moves the kernel
> forwards.

I agree.

thanks,

greg k-h
--


From: Luck, Tony <tony.luck@...>
Subject: RE: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 7:23 pm 2008

> My raw numbers show that the number of individual kernel contributors
> continues to increase with every release, so this might not be as much
> of a problem as it's made out to be.

That depends on whether we are gradually adding to the pool
of developers, or seeing an increasing stream of newcomers
who supply a patch or two before disappearing again.

If you look at the list of contributors some old release for
which we have good data (say 2.6.16).  How many of those people
contributed to each of the following releases? Does the
decay curve look steeper or more gentle if you start from
a more recent release?

-Tony
--


From: Greg KH <greg@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 8:36 pm 2008

On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
> > My raw numbers show that the number of individual kernel contributors
> > continues to increase with every release, so this might not be as much
> > of a problem as it's made out to be.
> 
> That depends on whether we are gradually adding to the pool
> of developers, or seeing an increasing stream of newcomers
> who supply a patch or two before disappearing again.
> 
> If you look at the list of contributors some old release for
> which we have good data (say 2.6.16).  How many of those people
> contributed to each of the following releases? Does the
> decay curve look steeper or more gentle if you start from
> a more recent release?

I don't know, I haven't tracked the people individually that way, only
looked at the basic numbers of developers per release.

thanks,

greg k-h
--


From: Dave Jones <davej@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 9:00 pm 2008

On Wed, May 28, 2008 at 05:36:57PM -0700, Greg Kroah-Hartman wrote:
 > On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
 > > > My raw numbers show that the number of individual kernel contributors
 > > > continues to increase with every release, so this might not be as much
 > > > of a problem as it's made out to be.
 > > 
 > > That depends on whether we are gradually adding to the pool
 > > of developers, or seeing an increasing stream of newcomers
 > > who supply a patch or two before disappearing again.
 > > 
 > > If you look at the list of contributors some old release for
 > > which we have good data (say 2.6.16).  How many of those people
 > > contributed to each of the following releases? Does the
 > > decay curve look steeper or more gentle if you start from
 > > a more recent release?
 > 
 > I don't know, I haven't tracked the people individually that way, only
 > looked at the basic numbers of developers per release.

Are you just doing something like
 git log v2.6.16..v2.6.17 | grep ^Author: | sort -u| wc -l  

or do you have some script that maps addresses to people ?
One person may appear once in 2.6.20, and then a half dozen times
in 2.6.21 if they use multiple email addresses for example.
(Also, typos, and people using full hostnames in their sign-off's
 instead of email addresses skew this somewhat).

I'm guessing the latter, due to the graph thing you did. Pointers?

	Dave

-- 
http://www.codemonkey.org.uk
--


From: Greg KH <greg@...>
Subject: Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project
Date: May 28, 10:26 pm 2008

On Wed, May 28, 2008 at 09:00:55PM -0400, Dave Jones wrote:
> On Wed, May 28, 2008 at 05:36:57PM -0700, Greg Kroah-Hartman wrote:
>  > On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
>  > > > My raw numbers show that the number of individual kernel contributors
>  > > > continues to increase with every release, so this might not be as much
>  > > > of a problem as it's made out to be.
>  > > 
>  > > That depends on whether we are gradually adding to the pool
>  > > of developers, or seeing an increasing stream of newcomers
>  > > who supply a patch or two before disappearing again.
>  > > 
>  > > If you look at the list of contributors some old release for
>  > > which we have good data (say 2.6.16).  How many of those people
>  > > contributed to each of the following releases? Does the
>  > > decay curve look steeper or more gentle if you start from
>  > > a more recent release?
>  > 
>  > I don't know, I haven't tracked the people individually that way, only
>  > looked at the basic numbers of developers per release.
> 
> Are you just doing something like
>  git log v2.6.16..v2.6.17 | grep ^Author: | sort -u| wc -l  

Ah, if it were only so easy :)

> or do you have some script that maps addresses to people ?
> One person may appear once in 2.6.20, and then a half dozen times
> in 2.6.21 if they use multiple email addresses for example.
> (Also, typos, and people using full hostnames in their sign-off's
>  instead of email addresses skew this somewhat).
> 
> I'm guessing the latter, due to the graph thing you did. Pointers?

Yes, it's the later.  Jon Corbet has a great little python tool that we
have used to create the "who is writing the kernel" series of articles.
I've been using it to keep track of who maps to what email address and
for what company for a while now.

An older version can be found at:
	http://www.kernel.org/pub/linux/kernel/people/gregkh/kernel_history/
it's the 'gitdm' tool there.  I don't know if he has an updated version
around anywhere, I suppose as it looks like I'm doing the releases for
it, I can put up a new snapshot if people are really interested.

I also have "cleaned up" versions of the kernel log files for just the
reason you say above.  You would not believe the number of times some
people mispell their own name in a single kernel release...  That makes
it easier to do this kind of mapping.  The cleaned up logs are in that
directory as well.

thanks,

greg k-h
--