login
Search
Keith Owens recently announced the 3.0 release of kbuild 2.5. Major changes since release 2.0 [earlier story] include replacing the mdbm database engine with a custom engine for a performance increase, and removal of CML2 support. He followed up the release with some build time statistics comparing the existing build system to kbuild 2.5, saying, "I don't care how fast a build is, if it is not accurate then the time is wasted. Fortunately kbuild 2.5 is both fast and 100% accurate, unlike the existing build system."
In another thread, Linux creator and 2.5 kernel maintainer Linus Torvalds acknowledged that for now he was only accepting kbuild patches from Kai Germaschewski [earlier story], including those for kbuild 2.5. He offered several reasons, including the desire to avoid "flag day" patches preferring instead gradual merges of which Kai has demonstrated skill, that Kai is familiar with the build system, and finally that Kai is not a kbuild 2.5 supporter, "Which is a plus in my book: it means that whatever Kai tries to push my way I'll feel just that much more comfortable with as having had critical review."
From: Keith Owens Subject: Announce: Kernel Build for 2.5, release 3.0 is available Date: Mon, 03 Jun 2002 12:35:05 +1000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-Type: text/plain; charset=us-ascii Release 3.0 of kernel build for kernel 2.5 (kbuild 2.5) is available. http://sourceforge.net/projects/kbuild/, package kbuild-2.5, download release 3.0.kbuild-2.5-core-15
Changes from core-14.
Replace mdbm with kbuild specific database engine to increase
performance.
Remove CML2 support, Dominik Brodowski, Keith Owens.
Remove the restriction on symlinked sources and targets. Aegis
users should be able to use kbuild 2.5 now.
kbuild-2.5-common-2.5.19-1.
Changes from common-2.5.15-4.
Upgrade to kernel 2.5.19.
kbuild-2.5-i386-2.5.19-1.
Changes from i386-2.5.15-2.
Upgrade to kernel 2.5.19.Larry McVoy provided the mdbm database engine from BitKeeper for use in
kbuild 2.5 release 2.0-2.4, and was extremely helpful in answering my
questions about mdbm. Unfortunately some of the processing required by
kbuild 2.5 did not fit well with the mdbm model. Since an application
specific database will always outperform a generic database, I took
advantage of knowledge about the kbuild data and wrote an application
specific database engine.This is not a criticism of mdbm - it is a good generic database engine.
I could squeeze a few more seconds out of the build time by using
application specific knowledge.-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: Exmh version 2.1.1 10/15/1999
iD8DBQE8+tXYi4UHNye0ZOoRAp+5AJ9evi07/7D1+xHRRnxA8xdpeqY3hgCeMcrH
kgNAIKaGHE1Fy8BKByBZqXs=
=EamL
-----END PGP SIGNATURE-----From: Hayden James
Subject: Re: Announce: Kernel Build for 2.5, release 3.0 is available
Date: Mon, 3 Jun 2002 00:06:16 -0400Why was CML2 support removed?
From: Keith Owens
Subject: Re: Announce: Kernel Build for 2.5, release 3.0 is available
Date: Mon, 03 Jun 2002 14:14:24 +1000On Mon, 3 Jun 2002 00:06:16 -0400, Hayden James wrote:
>Why was CML2 support removed?ESR has dropped off the list. CML2 and kbuild 2.5 are completely
independent and having the two in the same patch was getting messy.
The config rules in kbuild 2.5 are clean, support for other variants of
CML can be added at any time.
From: Keith Owens
Subject: kbuild 2.5 timing comparisons
Date: Mon, 03 Jun 2002 12:58:46 +1000Timing comparisons of kbuild 2.5 release 3.0 and existing 2.5.19
makefiles.======================================================================
I don't care how fast a build is, if it is not accurate then the time
is wasted. Fortunately kbuild 2.5 is both fast and 100% accurate,
unlike the existing build system.======================================================================
Full 2.5.19 config. The full config is too big for bzImage so build vmlinux.
Machine 1 - 4 x PIII @700 MHz, 1GB ram.kbuild 2.5
make -f Makefile-2.5 -j4 allyes installable 14:11.39
make -f Makefile-2.5 -j4 00:07.94 [a]
rm drivers/net/wan/cosa.o (module)
make -f Makefile-2.5 -j4 00:10.79
rm drivers/net/wan/cosa.o
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:06.36 [b]
rm drivers/net/wan/cosa.o
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:04.51 [c]
rm vmlinux
make -f Makefile-2.5 -j4 00:41.80 [d]
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:03.48 [b]
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:01.63 [e]
[a] No spurious rebuilds in kbuild 2.5. 8 seconds to regenerate
the global makefile and determine that the entire kernel is up
to date.
[b] First use of NO_MAKEFILE_GEN=1 takes ~2 seconds to set up the
correct environment.
[c] Mainly the build time for cosa.o.
[d] Mainly the link time for vmlinux.
[e] 1.6 seconds to run the entire kernel makefile once it has been
built.
Existing kbuild. Slightly shorter .config because some of the 2.5.19
Makefiles are broken. The code compiles under kbuild 2.5 but not
under the 2.5.19 makefiles.
make oldconfig dep 00:55.21 [a]
make -j4 BUILD_MODULES=1 vmlinux 13:48.89
make -j4 BUILD_MODULES=1 vmlinux 04:51.24 [b]
make -j4 BUILD_MODULES=1 vmlinux 00:48.20 [c]
make -j4 BUILD_MODULES=1 vmlinux 00:43.15 [d]
[a] make dep is not parallel safe.
[b] No change from [a], but spurious rebuilds all over the place.
That's what you get for using recursive make instead of a
global makefile.
[c] No change from [b], but it still has spurious rebuilds.
[d] No change from [c], it rebuilt init files and relinked vmlinux
for no good reason.
Not only is the existing build system significantly slower than kbuild
2.5, it is also unreliable. It takes four builds to get a stable
result.
Total time to a stable result:
kbuild 2.5 14:11 (one pass)
Existing 21:00+ (four passes)
======================================================================
Minimal 2.5.19 config.
Machine 2 - 1 x PIII @550 MHz, 32MB ram.
kbuild 2.5
make -f Makefile-2.5 allno installable 03:53.16
make -f Makefile-2.5 00:19.89 [a]
rm vmlinux
make -f Makefile-2.5 00:26.08
make -f Makefile-2.5 NO_MAKEFILE_GEN=1 00:04.31 [b]
make -f Makefile-2.5 NO_MAKEFILE_GEN=1 00:01.38 [c]
[a] No spurious rebuilds in kbuild 2.5.
[b] First use of NO_MAKEFILE_GEN=1 takes ~2 seconds to set up the
correct environment.
[c] 1.38 seconds to run the entire kernel makefile once it has been
built.
Existing kbuild on same config.
make oldconfig dep 01:10.72
make BUILD_MODULES=1 vmlinux 02:43.88
make BUILD_MODULES=1 vmlinux 00:07.04
On a minimal configuration, the existing system is slightly faster.
But that comes at the expense of an unreliable build method on
parallel builds.
======================================================================
Medium 2.5.19 config.
Machine 2 - 1 x PIII @550 MHz, 32MB ram.
kbuild 2.5
make -f Makefile-2.5 oldconfig installable 10:26.88
make -f Makefile-2.5 00:27.58
Existing kbuild.
make BUILD_MODULES=1 oldconfig dep vmlinux 10:18.09
On a small machine, the two systems are comparable. But only kbuild
2.5 is accurate.
2:01.12
0:05.55
Medium 2.5.19 config.
Machine 1 - 4 x PIII @700 MHz, 1GB ram.
kbuild 2.5
make -f Makefile-2.5 oldconfig installable 02:01.12
make -f Makefile-2.5 00:05.55
Existing kbuild.
make oldconfig dep 00:44.87 [a]
make -j4 BUILD_MODULES=1 vmlinux 01:43.06
make -j4 BUILD_MODULES=1 vmlinux 00:36.28 [b]
make -j4 BUILD_MODULES=1 vmlinux 00:03.79 [c]
[a] make dep is not parallel safe.
[b] No change from [a], but spurious rebuilds all over the place.
That's what you get for using recursive make instead of a
global makefile.
[c] No change from [b], finally stable.
kbuild 2.5 is both faster and accurate.======================================================================
The spurious rebuilds are not even deterministic. They depend on the
relative time that each directory is processed (which depends on the
system load and the number of processors) and the timestamps on the
files before you did the build. IOW, the spurious rebuilds depend on
what you did in the source tree last time.Which raises the interesting question - how many of the rebuilds are
spurious and how many are really required?To demonstrate the unreliable parallel build on an SMP box -
find -type f | xargs touch
make oldconfig dep
make -j4 BUILD_MODULES=1 vmlinux
make -j4 BUILD_MODULES=1 vmlinuxDon't be surprised if it rebuilds vmlinux even if nothing else changes.
There is a bug somewhere in the existing 2.5.19 makefiles that does not
get the rebuild rule right.======================================================================
The fact that the existing rules generate spurious rebuilds is bad
enough. What is much worse is that they do not detect changes
correctly, so objects are not recompiled when they should be. Build a
config with all of ACPI turned on, run the existing rules until you get
a stable build (all spurious rebuilds have finished).echo '#warning aclocal.h used' >> drivers/acpi/include/aclocal.h
make -j4 BUILD_MODULES=1 vmlinuxACPI does not rebuild, even though its headers have changed.
======================================================================
I don't care how fast a build is, if it is not accurate then the time
is wasted. Fortunately kbuild 2.5 is both fast and 100% accurate,
unlike the existing build system.From: Keith Owens
Subject: Re: kbuild 2.5 timing comparisons
Date: Mon, 03 Jun 2002 15:48:29 +1000Timing comparisons of kbuild 2.5 release 3.0 and existing 2.5.20
makefiles.======================================================================
I don't care how fast a build is, if it is not accurate then the time
is wasted. Fortunately kbuild 2.5 is both fast and 100% accurate,
unlike the existing build system.======================================================================
Full 2.5.20 config. The full config is too big for bzImage so build
vmlinux. This is a smaller config than 2.5.19, 2.5.20 introduced yet
more driver errors so fewer objects will build.
Machine 1 - 4 x PIII @700 MHz, 1GB ram.kbuild 2.5
make -f Makefile-2.5 -j4 allyes installable 13:43.53
make -f Makefile-2.5 -j4 00:07.94 [a]
rm drivers/net/wan/cosa.o (module)
make -f Makefile-2.5 -j4 00:10.72
rm drivers/net/wan/cosa.o
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:06.30 [b]
rm drivers/net/wan/cosa.o
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:04.43 [c]
rm vmlinux
make -f Makefile-2.5 -j4 00:37.21 [d]
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:03.47 [b]
make -f Makefile-2.5 -j4 NO_MAKEFILE_GEN=1 00:01.57 [e]
[a] No spurious rebuilds in kbuild 2.5. 8 seconds to regenerate
the global makefile and determine that the entire kernel is up
to date.
[b] First use of NO_MAKEFILE_GEN=1 takes ~2 seconds to set up the
correct environment.
[c] Mainly the build time for cosa.o.
[d] Mainly the link time for vmlinux.
[e] 1.6 seconds to run the entire kernel makefile once it has been
built.
Existing kbuild. Slightly shorter .config because some of the 2.5.20
Makefiles are broken. The code compiles under kbuild 2.5 but not
under the 2.5.20 makefiles.
make oldconfig dep 00:54.89 [a]
make -j4 BUILD_MODULES=1 vmlinux 13:10.45
make -j4 BUILD_MODULES=1 vmlinux 05:24.95 [b]
make -j4 BUILD_MODULES=1 vmlinux 00:39.92 [c]
make -j4 BUILD_MODULES=1 vmlinux 00:10.43 [d]
[a] make dep is not parallel safe.
[b] No change from [a], but spurious rebuilds all over the place.
That's what you get for using recursive make instead of a
global makefile.
[c] No change from [b], but it still has spurious rebuilds.
[d] Finally a stable build.
Not only is the existing build system significantly slower than kbuild
2.5, it is also unreliable. It takes four builds to verify that you
have a stable result.
Total time to a stable result:
kbuild 2.5 13:43 (one pass)
Existing 20:00+ (four passes)======================================================================
The spurious rebuilds are not even deterministic. They depend on the
relative time that each directory is processed (which depends on the
system load and the number of processors) and the timestamps on the
files before you did the build. IOW, the spurious rebuilds depend on
what you did in the source tree last time.Which raises the interesting question - how many of the rebuilds are
spurious and how many are really required?To demonstrate the unreliable parallel build on an SMP box -
find -type f | xargs touch
make oldconfig dep
make -j4 BUILD_MODULES=1 vmlinux
make -j4 BUILD_MODULES=1 vmlinux======================================================================
The fact that the existing rules generate spurious rebuilds is bad
enough. What is much worse is that they do not detect changes
correctly, so objects are not recompiled when they should be. Build a
config with all of ACPI turned on, run the existing rules until you get
a stable build (all spurious rebuilds have finished).echo '#warning aclocal.h used' >> drivers/acpi/include/aclocal.h
make -j4 BUILD_MODULES=1 vmlinuxACPI does not rebuild, even though its headers have changed.
======================================================================
I don't care how fast a build is, if it is not accurate then the time
is wasted. Fortunately kbuild 2.5 is both fast and 100% accurate,
unlike the existing build system.
From: Dan Kegel
Subject: Re: KBuild 2.5 Impressions
Date: Sun, 02 Jun 2002 19:49:47 -0700Daniel Phillips wrote:
> Well, actually a lot of the work done by Kai is simply importing
> portions of Keith's work that break out easily, which is purely
> duplication of effort, since such work is already in progress. In
> fact it creates more work, because then we have to go parse Kai's
> patches and find out what he submitted, then see if it gets applied
> so we can mark it 'applied' in the list. This is a real waste of
> time, and did I mention, it's divisive?Linus sees Kai as being the most promising fellow to
integrate kbuild2.5 right now (see
http://marc.theaimsgroup.com/?l=linux-kernel&m=102307114005894&w=2 )
and Kai is willing to take it on in just the way Linus wants.
It's probably worth giving Kai and Linus the benefit of the doubt for a while,
even if it does mean having to rejigger the kbuild-2.5 patch
each time Linux accepts one of Kai's patches.I personally am anxious to see kbuild-2.5 make it into the kernel,
but I also feel it can only benefit from a strong review of
the sort that comes about during gradual evolution
of the kernel build process towards the techniques used by kbuild-2.5.Thanks to everyone, Keith, Daniel, Thunder, and Kai, who are
working (together or not!) on moving the kernel build process
into the modern era.
- DanFrom: Linus Torvalds
Subject: Re: KBuild 2.5 Impressions
Date: Mon, 3 Jun 2002 03:28:37 +0000 (UTC)In article Dan Kegel wrote:
>
>Linus sees Kai as being the most promising fellow to
>integrate kbuild2.5 right now [...]Side note, just to explain _why_ I prefer it done this way, so that
people can understand - even if they don't necessarily have to agree
with - why this is my preferred approach.There's actually several reasons:
- I always hate "flag day" patches. Do they happen? Sure. Some people
have already given examples of such big flag-day patches, the ALSA
merge being one prime example. That doesn't mean that I like them
any more for that.
In short: if at all possible, I _much_ prefer gradual merges, where
"gradual" really means that features are added one-by-one (and that
does _not_ mean "build up the infrastructure slowly, so that the
final 'flag-day' patch itself is small but has large ramifications")
- Kai has already shown that he can merge with me easily, and actually
took one traditional flag-day-project (ISDN: every single merge was a
flag-day merge), and has turned that into a very easy gradual merge
for me. I used to dread ISDN merges, these days I don't even have to
think about them.
- Kai obviously already knows the build system, as he has been doing a
lot of incremental stuff on it already.
- Kai isn't an enthusiastic kbuild-2.5 supporter. In fact, he tends to
be a bit down on some of it. Which is a plus in my book: it means
that whatever Kai tries to push my way I'll feel just that much more
comfortable with as having had critical review.So let's see how it works out. Maybe it won't, but this would seem
workable at least in theory.Linus
Linus is annoying.
How the hell do you gradually merge kbuild 2.5?
It's a completely new build system, which has nothing to do with the old one?
And it is superior in every single way.
Kai doesn't even know why he doesn't like it. He just doesn't, which seems to be the common theme in the shooting doing of CML2/kbuild 2.5 etc.
And the thing can run parallel to the existing system. So all the moronic nay-sayers can gradually switch over if they prefer.
It seems to me that Keith should be in charge of the Linux build system, not Kai. Very annoying. ARGH!
You are completely right, and this is how it has to be
What you have identified is a classic element of the global life cycle.
Way back in the 60's there were established ways of running things, and some people decided there was A Better Way, so they started the MULTICS project. This project got bogged down and some of its participants decided there was A Better Way, and started UNIX. Then UNIX got bogged down and Berkley and a few other institutions decided there was A Better Way and that turned into BSD. Then BSD got bogged down. There were Xenix, Minix, and then Linux. It happens over and over.
See "The Lucifer Principle" for more examples and explinations of how it fits into the cycle of life.
The Big Question you're implying is, why doesn't the organism continue to grow the way it did when it was young? The reason is that it has passed that part of its life cycle. You'll see references to it in everything from astrology to zoology: the organism starts fragile but adaptable, quick, learning, forming itself, molding itself. It establishes itself, then it digs in, spawns offspring, decays and goes away. Details change from case to case, but the overall pattern repeats like the patterns we see in leaves, branches, veins, rivers, and the immaginary trees we create to organize ourselves around projects.
What can we learn from this? What do we do with this self-awareness? There are two conclusions you can come to. Either the organism is still growing and just needs more prodding, or it has begun the phases where it prepares the world for the next generation. In the former case, you should encourage the organism to persue the avenue you think is best for it. In the latter case, you should seek out the offspring and encourage them to pursue the path which the parent organism doesn't have the life-force left to expend on.
Give them a break
Seems to me lots of people are assuming that Kai and Linus look for any reason not to change anything. The way I read lkml, that does not seem to be true.
linus on lkml
answer from Kai
/jarek
No Linus is a bastard, but you should of known that already
Linus himself says he's a bastard people seem to forget this when their favorite patch/project/whatever gets rejected by him. He does give clues however and having seen him talk about avoiding "flag day" merges I can see exactly where he is coming from.
Having managed a few projects myself in my time I know the problems you can introduce with a "everyone stop and pick up the tree in two days" approach. If it caused me problems when I was only dealing with 10 engineers think what it will do to the mass of linux developers.
As far as I can see Linus's approach seems, dare I say it, pragmatic.
Alex
two words
Institutional Inertia
that's sillyness
Linus is hardly an institution all by himself.
Honestly, I have never seen anyone who is more open to change than Linus. Take a look at the Daleki patches. Linus stood up for them at times when no one else would. (It turned out to be the right decision incidentally).
Because of the long 2.4 series there is a back log of patches. Never before have so many changes been going into the kernel. It's sort of silly to say things about institutional intertia when the kernel is changing faster than ever before don't you think?
Good
KBuild will make it in, after a tough review by Kai. Linus will get the patches in the format he likes. The key line is right here:
Kai isn't an enthusiastic kbuild-2.5 supporter. In fact, he tends to be a bit down on some of it. Which is a plus in my book: it means that whatever Kai tries to push my way I'll feel just that much more comfortable with as having had critical review.
Linus, has indicated he wants kbuild-2.5 in. It may take longer then anyone likes, but it will be in 2.5.