Recent 2.5 development kernel releases have included a string of kbuild 2.4 fixes by Kai Germashewski. For example, yesterday's release of 2.5.19 included a list of 13 kbuild patches. In a recent email, Kai offered an explanation of his recent efforts, "There is still quite a bit left to do (in particular improving dependency generation and modversions handling), but I think it makes sense to explain what happened so far."

In short order, the subject of kbuild 2.5 was brought up [earlier story]. Kai aknowledged kbuild 2.5 saying, "I believe kbuild-2.5 has lots of useful ideas and I'll go pick pieces". Linus evidently will not merge kbuild 2.5 until it is broken into smaller pieces, similar to his earlier request for CML2. Kai expressed his agreement, "Splitting things would at least allow for discussing whether these (and other) individual features are considered worthwhile or not, as opposed to getting the full bag or nothing."

Much of the recent thread follows.

From: Kai Germaschewski
To: linux-kernel mailing list
Subject: 2.5.19 - What's up with the kernel build?
Date: 	Wed, 29 May 2002 15:22:52 -0500 (CDT)

Okay, as people probably already noticed, I touched quite a lot of
Makefiles lately. Contrary to what the patches (which touched many
files) might imply, not much did actually change, in particular not
much visible outside of the kernel build process.

There is still quite a bit left to do (in particular improving
dependency generation and modversions handling), but I think it makes
sense to explain what happened so far.

There's also some points (marked with >>>) where I'd like to get
feedback on how things should be handled in the future.


For users (i.e. people who compile kernels)
-------------------------------------------
Not much changed at all. Do what you always did, it should still work
the same.

o vmlinux/(b)zImage etc. and modules can now be built in the same
  pass.
  For testing this feature, please use

      make BUILD_MODULES=1 vmlinuz/(b)zImage/whatever,

  i.e. add the "BUILD_MODULES=1" to the command line you normally use,
  and you save the additional "make modules" run.

>>> What do you guys think how this should be handled in the future? -
>>> my suggestion would be to make this behavior the default, and, in
>>> case someone needs it, add the option to say "BUILD_MODULES=0".
  
o you can ask to build a single target, like 

      make some/dir/foo.[iso]

  and it'll give you preprocessed/assembler/object file output, provided
  that some/dir/foo.c or some/dir/foo.S exists. It should get all cases
  correct (well, except for arch/i386/kernel/{head.o,init_task.o}, which
  will cause an entire recursive run), with adding the right flags,
  compiling as modular etc.

o when your source tree changed only little or not at all, performance
  should be quite a bit improved. If you do "make BUILD_MODULES=1 
  vmlinux", the number of "make" invocations should be down by about a 
  factor of three compared to a previous "make vmlinux modules". Also, 
  vmlinux will not be relinked if nothing actually changed. (If lots of 
  files changed, the time make consumes will be down in the noise, the 
  build time will be dominated by gcc).

>>> It's possible with only small changes to provide a quiet mode now,
>>> which would not print the entire command lines but only
>>>
>>>	  Descending into drivers/isdn/kcapi
>>>	  Compiling kcapi.o
>>>	  Compiling capiutil.o
>>>	  Linking kernelcapi.o
>>>	  ...
>>>
>>> Is that considered useful?

For developers:
---------------

If you had clean Makefiles for your subsystem/driver/whatever before,
again, not much changed.

o "list-multi" + associated link rules are gone for good. It's been
  like that for a couple of months now, so that's probably no news
  for you.

o Descending into and linking subdirs was often handled with constructs
  like the following:

       subdir-$(CONFIG_E100) += e100
       ifeq ($(CONFIG_E100),y)
         obj-y += e100/e100.o
       endif 

  This can now be expressed in a much more compact way as

       obj-$(CONFIG_E100) += e100/

  which basically means the exact same thing:
  - if CONFIG_E100=y, descend into the subdir and link the result
  - if CONFIG_E100=m, descend into the subdir and build the modules there
  - otherwise, ignore that subdir

  Only requirement is that the O_TARGET in the subdir (that's what getting
  linked in, "e100.o" in the example above) is named "built-in.o". To
  simplify things even further, just leaving out the "O_TARGET := ..."
  line does the right thing, it uses "built-in.o" as default. Anyway,
  the conversion is mostly done now, so you don't have to actually deal
  with it.

o Once we make "module and built-in in one pass" default, the

       mod-subdirs := ...

  lines can go, too. Nobody understood what they're there for anyway,
  right? ;-)

For arch maintainers:
---------------------

o In the arch specific Makefiles, you needed to build not only 
  O_TARGET (or L_TARGET) out of $(obj-y) and the modules 
  in $(obj-m), but also additional objects, e.g. for i386: 
  "head.o" and "init_task.o"

  This can (and should) be done cleanly now, by adding such targets to
  the variable EXTRA_TARGETS instead of playing games with overriding
  Rules.make's "first_rule:"

  The conversion is basically done, so you don't really need to
  bother.

o Rules.make provides standard rules for assembling files now,

  So use them - again, the conversion is done already, just don't undo
  it.

BTW, yeah, I know, Documentation/kbuild/makefiles.txt could use an

update. Will do that...
--Kai
From: Paul P Komkoff Jr

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Thu, 30 May 2002 00:50:05 +0400
Replying to Kai Germaschewski:

> There is still quite a bit left to do (in particular improving

> dependency generation and modversions handling), but I think it makes

> sense to explain what happened so far.

>

> There's also some points (marked with >>>) where I'd like to get

> feedback on how things should be handled in the future.
May I ask you just one questin?

Have you read (yet) kbuild25 ?
Ahh, no, another one.

Is this a "signs" of kbuild25 being thrown away like cml2 ?

If yes, then I am very unhappy person now (c) ac
Played with kbuild25 today I migrated 2.4.19-pre8-ac5-s2 to it. Now make -f

Makefile-2.5 is a preferred way to make here. New system is much cleaner,

and don't need that mess of makedep and listmultis. But hey, people you

should already know that whilst you trying to install a couple of new nails

to rotten construct to help it stay for another couple of time intervals ...
:(((
From: Jeff Garzik

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 16:58:32 -0400
Well, I really like Keith's kbuild25 too, but Linus said (at least once)

he wanted an evolution to a new build system... not an unreasonable

request to at least consider.  Despite Keith's quality of code (again --

I like kbuild25), his 3 patch submissions seemed a lot like ultimatums,

very "take it or leave it dammit".  Not the best way to win friends and

influence people.
If Keith is indeed leaving it, I'm hoping someone will maintain it, or

work with Kai to integrate it into 2.5.x.
    Jeff
From: Thunder from the hill

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 16:56:26 -0600 (MDT)
Hi,
On Wed, 29 May 2002, Nicolas Pitre wrote:

> So it looks like someone else will have to volunteer to split kbuild25 into

> multiple small patches and feed them "piecemeal" to Linus before we ever see

> it into the kernel tree.
Well, I would, but... wasn't Kai on that already?
Regards,

Thunder
From: Nicolas Pitre

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 19:18:57 -0400 (EDT)
On Wed, 29 May 2002, Thunder from the hill wrote:

>

> Well, I would, but... wasn't Kai on that already?
No.
Kai is only banging on the current build system to make it somewhat more

palatable.  While this is certainly useful today, those efforts would

probably produce a better system in the long run if they were directed at

kbuild25 instead.
Or maybe people just don't care enough about the build system for kbuild25

to be worth it...
Nicolas
From: Daniel Phillips

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Thu, 30 May 2002 01:29:01 +0200
On Thursday 30 May 2002 01:18, Nicolas Pitre wrote:

> Or maybe people just don't care enough about the build system for kbuild25

> to be worth it...
Omigod, don't even think that.  Kbuild 2.5 is faster and better than the

current kbuild.  I, for one, am waiting - impatiently - for the thing to get

in the tree.  The current build system is slow and unreliable, and don't even

think of trying to build two different architectures in the same source tree

at the same time.
--

Daniel

From: Nicolas Pitre

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 18:38:27 -0400 (EDT)
On Wed, 29 May 2002, Jeff Garzik wrote:

> Well, I really like Keith's kbuild25 too, but Linus said (at least once)

> he wanted an evolution to a new build system... not an unreasonable

> request to at least consider.  Despite Keith's quality of code (again --

> I like kbuild25), his 3 patch submissions seemed a lot like ultimatums,

> very "take it or leave it dammit".  Not the best way to win friends and

> influence people.

>

> If Keith is indeed leaving it, I'm hoping someone will maintain it, or

> work with Kai to integrate it into 2.5.x.
When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith)

considered that was a "stupid comment" and that he'd ignore stupid comments.
So it looks like someone else will have to volunteer to split kbuild25 into

multiple small patches and feed them "piecemeal" to Linus before we ever see

it into the kernel tree.
Nicolas
From: Tomas Szepe

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Thu, 30 May 2002 01:06:57 +0200
> When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith)

> considered that was a "stupid comment" and that he'd ignore stupid comments.
What remains to be answered is, how does one split a system of a myriad of

build rule files into a reasonable amount of small patches.
Of course, you could have hundreds of patches each consisting of a single

Makefile.in, but how would that make the reviewing/integrating easier? In

the end you'd end up reading the same input, only you'd complement it by

frequently pressing your favorite show-me-the-next-mail key.
The solution here is not to create "artificial splitting," but rather spare

the good ol' system for the time being and have kbuild25 coexist with it until

all build issues are resolved and everything works acceptably -- and that's

what has been offered. 2.5 is certain to span a long enough period for such

treatment, and Keith will be so kind as to keep updating his work.
Alright I really didn't want to get involved in another kbuild thread but

couldn't help it. Sorry.
T.

From: Daniel Phillips

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Thu, 30 May 2002 01:40:46 +0200
On Thursday 30 May 2002 01:06, Tomas Szepe wrote:

> > > Well, I really like Keith's kbuild25 too, but Linus said (at least once)

> > > he wanted an evolution to a new build system... not an unreasonable

> > > request to at least consider.  Despite Keith's quality of code (again --

> > > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums,

> > > very "take it or leave it dammit".  Not the best way to win friends and

> > > influence people.
OK that's true, but think: how much work has Keith put into this?  How much

did he or his employer get paid?  And how many times has he been told to go

off and fix something, as a prelude to getting the thing in?  The last time

it was the first-time build speed.  He went away and came back with a *huge*

improvement, even more than what people were demanding.  You'd think that

would be enough.
Keith has to do *two* full time jobs as long as the patch isn't merged:

developing the patch itself and tracking the whole 2.5 tree as it (rapidly)

evolves.
> > > If Keith is indeed leaving it, I'm hoping someone will maintain it, or

> > > work with Kai to integrate it into 2.5.x.

> >

> > When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith)

> > considered that was a "stupid comment" and that he'd ignore stupid comments.
It is of course always regrettable when one is so rash as to call a stupid

comment a stupid comment ;-)
> What remains to be answered is, how does one split a system of a myriad of

> build rule files into a reasonable amount of small patches.

>

> Of course, you could have hundreds of patches each consisting of a single

> Makefile.in, but how would that make the reviewing/integrating easier? In

> the end you'd end up reading the same input, only you'd complement it by
> frequently pressing your favorite show-me-the-next-mail key.
I thought BitKeeper was supposed to be able to deal with precisely this sort

of merge problem.  In this case, splitting the thing up just seems

unnatural, and a dubious use of time.
--

Daniel
From: Nicolas Pitre

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 20:17:58 -0400 (EDT)
On Thu, 30 May 2002, Daniel Phillips wrote:
> Omigod, don't even think that.  Kbuild 2.5 is faster and better than the

> current kbuild.  I, for one, am waiting - impatiently - for the thing to get

> in the tree.  The current build system is slow and unreliable, and don't even

> think of trying to build two different architectures in the same source tree

> at the same time.
I totally agree with you.
Then let me rephrase it that way: maybe not enough people care enough about

the build system for kbuild25 to be worth it...
> > > > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums,

> > > > very "take it or leave it dammit".  Not the best way to win friends and

> > > > influence people.

>

> OK that's true, but think: how much work has Keith put into this?  How much

> did he or his employer get paid?  And how many times has he been told to go

> off and fix something, as a prelude to getting the thing in?  The last time

> it was the first-time build speed.  He went away and came back with a *huge*

> improvement, even more than what people were demanding.  You'd think that

> would be enough.
But is it really enough?
Linus himself once said: "Especially as I don't find the existign system so

broken." He's not alone according to the amount (or lack) of public

complains with regards to the current system.
> I thought BitKeeper was supposed to be able to deal with precisely this sort

> of merge problem.  In this case, splitting the thing up just seems

> unnatural, and a dubious use of time.
Indeed, I agree.
But until there is enough people committed to kb25 to the point of pushing

for its wider adoption, either by releasing traditional kernel patches (like

the -dj, -aa, -ac or whatever) actually carrying kb25 so to convince Linus

and others, or producing "piecemeal" patches so it can be merged by Linus

with less resistance, well nothing will hapen.
If people aren't interested enough and/or willing to comply with Linus'

requirements this will be a sad dead end, regardless the amount of effort

Keith put into this.
Nicolas
From: Daniel Phillips

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Thu, 30 May 2002 02:36:39 +0200
On Thursday 30 May 2002 02:17, Nicolas Pitre wrote:

> But is it really enough?

>

> Linus himself once said: "Especially as I don't find the existign system so

> broken." He's not alone according to the amount (or lack) of public

> complains with regards to the current system.
I think a lot of us were not complaining because we thought it was a done deal.

Now what's necessary, do we have to form cheerleading teams and start lobbying

to get what we thought was coming anyway?  It's really not a pleasant feeling

to be put in that position.  Nobody relishes the role of outspoken advocate of

this or that, at least I do not.  I'd *much* rather be hacking.
> Indeed, I agree.

>

> But until there is enough people committed to kb25 to the point of pushing

> for its wider adoption, either by releasing traditional kernel patches (like

> the -dj, -aa, -ac or whatever) actually carrying kb25 so to convince Linus

> and others, or producing "piecemeal" patches so it can be merged by Linus

> with less resistance, well nothing will hapen.
You've got it backwards.  The function of the maintainers you mentioned is

to *resist* change, not promote it.  In this case, change comes from the top,

and as long as Linus doesn't realize that there really is a lot of demand

for this change, nothing is going to happen.
> If people aren't interested enough and/or willing to comply with Linus'

> requirements this will be a sad dead end, regardless the amount of effort

> Keith put into this.
Now wait, it seems to me that only Keith is being asked to comply with

requirements, and they're tough requirements.  The burden is just not

being shared equally.
--

Daniel

From: Jeff Garzik

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 20:42:53 -0400
One thing I would like to see is building where builddir != srcdir.
I don't particularly care if this is optional or made a requirement, but

it's darned useful...
From: Robert Love

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	29 May 2002 17:47:18 -0700
I second that - it is incredibly useful.
Next on my list would be fixing dependency issues.
	Robert Love

From: Kai Germaschewski

Subject: Re: 2.5.19 - What's up with the kernel build?

Date: 	Wed, 29 May 2002 21:00:33 -0500 (CDT)
On Wed, 29 May 2002, Jeff Garzik wrote:
> Well, I really like Keith's kbuild25 too, but Linus said (at least once)

> he wanted an evolution to a new build system... not an unreasonable

> request to at least consider.  Despite Keith's quality of code (again --

> I like kbuild25), his 3 patch submissions seemed a lot like ultimatums,

> very "take it or leave it dammit".  Not the best way to win friends and

> influence people.

>

> If Keith is indeed leaving it, I'm hoping someone will maintain it, or

> work with Kai to integrate it into 2.5.x.
Oh well, it really wasn't my intention to start the good old kbuild-2.5

thread at all.
Anyway, I believe kbuild-2.5 has lots of useful ideas and I'll go pick

pieces - from kbuild-2.5, from dancing-makefiles, from stuff I've done

myself and work on improving the current build system. But I believe in

make, and don't think I'll move away from it.
One thing these patches show is that gradual improvement is actually

possible, so far the kbuild process has gained quite some features with a

lot of small patches - and some bigger ones, but these are only trivial

cleanups.
Of course it happened that I introduced some bugs in the process, but the

fact that fixes were posted to linux-kernel by the next morning shows that

it's obviously possible for other people to grasp what's going on and fix

bugs. Rules.make is some 400 lines currently, that's quite a difference to

kbuild-2.5 core's 30000 lines of code.
Anyway, fortunately it's not up to me to decide what happens. From my

perspective the plan is to go on with this gradual improvement, in

particular

o fix dependencies / modversions (that includes "make dep" going away)

o allow for separate objdir (this one is actually easy for 95% of the

  compiled files which use standard rules, and lots of work for the

  remaining 5%. So it'll take time to remove the 5% special cases, after

  that things are pretty easy)
I believe that Keith will keep maintaining his kbuild-2.5 (if he isn't

willing to put work into it, that's definitely a reason not to merge it,

after all he's the only one who can fix bugs in it for all I can see).

At the end of the day we will see how the two systems compare and, who

knows, maybe my work will even have simplified getting kbuild-2.5 into the

kernel.
Unfortunately Keith, for all I can see, has an all-or-nothing attitude.

There's definitely things which could be broken out of kbuild-2.5,

like e.g.
o standardize asm-offsets.h generation

o make the build target a config option (I'm not actually sure if that

  is a good idea at all, using different targets for make seems a

  sensible concept to me, but anyway)
Splitting things would at least allow for discussing whether these (and

other) individual features are considered worthwhile or not, as opposed to

getting the full bag or nothing.
Also, getting back to the 95%, if you compare kbuild-2.5's Makefile.in's

and the Makefiles in 2.5.19, you will see that 95% of what's in there has

a one-to-one relationship between the two Makefiles. I can't teach "make"

to read the Makefile.in's which use an entirely new syntax, but since

kbuild-2.5 uses its owner parser, it could be adapted to understand the

old syntax (which expresses the exact same thing, so I don't really

understand why Keith didn't use the old format in the first place). That

would allow to drastically cut down the 20.000 lines Makefile.in diff and

would also simplify maintaining kbuild-2.5.
So that's my point of view.
--Kai