Linux: KBuild 2.5 Impressions

Submitted by nimrod
on May 30, 2002 - 8:02pm

For those who haven't tried kbuild-2.5 (myself included), and want to know how much of an improvement it offers over the old system, Daniel Phillips offered some hard statistics. Here are the results:

                         old kbuild       kbuild 2.5     Speedup
First time build: 7 min 8 sec 5 min 55 sec 17%
Incremental build: 25.94 sec 15.31 sec 41% :-)

Jeremy adds: Paul P Komkoff has picked up kbuild 2.5, and begun making available updated patches for the latest 2.4 kernel tree (x86 only so far). For those interested, he even created a kernel branch (-s) based on Alan Cox's -ac tree with kbuild 2.5 applied. He says, "Remember - I'm not Keith Owens - I'm just impressed by his work."

From: Daniel Phillips
Subject: KBuild 2.5 Impressions
Date: Thu, 30 May 2002 11:45:14 +0200

I wanted to know how well kbuild 2.5 really works, so I got the patches
from kbuild.sourceforge.net and gave them a test drive, comparing to
old kbuild. First I did an out-of-the-box build, then I did 'touch
fs/ext2/inode.c' and rebuilt. Results are as follows:

                         old kbuild       kbuild 2.5     Speedup

First time build: 7 min 8 sec 5 min 55 sec 17%
Incremental build: 25.94 sec 15.31 sec 41% :-)

The test system is a 2 x 1 GHz server with scsi raid. Note that I only
used one processor for the tests, as I normally do: in my experience,
with old kbuild I spend more time picking up the broken pices than
I save in compile time.

For good measure, I did a make mrproper and rebuild with -j3. That
completed in 3 minutes, 25 seconds. Very impressive, and what's more,
running that make a second time completes in 12 seconds, as opposed to
old make, which tends to erroneously rebuild a large number of files,
and sometimes does not complete the job correctly.

I have only one point on the curve, but it confirms what Keith has
been saying. The incremental build speed is especially important to
me. If I were working all the time on 2.5 - sadly, I'm not - I would
without question be using kbuild 2.5 simply by reason of the fast
incremental builds.

Miscellaneous Notes
-------------------

Along the way, old kbuild did the usual wrong things:

  - In the incremental build, 6 files rebuilt that should not have been

- Once, when I interrupted the make dep, subsequent make deps would
no longer work, forcing me to do make mrproper and start again.

- Way too much output to the screen

I took a quick look into some of the kbuild 2.5 Makefile.in files, just
to get a feeling for what they look like, and they look fine to me. For
example, fs/ext2 has acquired a Makefile.in file, with contents that
seem easily understandible. The syntax of these files is explained in
the documentation, which is provided by the patch set, in:

Documentation/kbuild/kbuild-2.5.txt

Reading through it, I found out about the NO_MAKEFILE_GEN, an option
that promises to make a fast build system even faster for interative
development, the type I always do. There is also a lot of informative
design documentation. Amazingly, there is even postscript
documentation. Full marks for documentation here.

Note to Keith: this file needs to be in some obvious place on the
sourceforge site.

Both the old and new build systems failed the rename test: rename the
root of the source tree and it will no longer build. The kbuild 2.5
documentation explains:

  Unlike kbuild 2.4, renaming a source or object tree does not force a
complete rebuild. After renaming any tree and updating the KBUILD_SRCTREE
and KBUILD_OBJTREE variables, just run make. Unless you have changed any
files, nothing should be rebuilt, however the make step must be run to save
the new tree names ready for the install step.

but this did not work for me. However, I do not doubt that it will
soon.

Reading through the patches, I noticed there is some useful looking
help text added for the makefile operations, but it wasn't immediately
obvious how to get it. Reading the patch is, of course, one way.

There is no Python anywhere to be seen in kbuild 2.5, for those who
worry about that. It is coded in C, about 10,000 lines it seems.
It has a simple built in database which I suppose accounts for some
of that. For what it does, it seems quite reasonable. There is a
bison-generated grammar, presumeably to parse the Makefile.in
language. This is the proper way do do things - much more reliable
and maintainable than a hand-coded parser, not to mention more
efficient. Most of the rest of the patch consists of Makefile.in
files. I can really see why this is a big job for Keith to maintain
by himself: the maintainers really need to be involved. I doubt that
it will be hard, and by appearances, there is no need for any
particular arch to step up to kbuild 2.5 immediately, since the old
build system still works. In any event, the 1386-specific part of
the patch is only 700 or so lines long.

The output that kbuild 2.5 generates is a vast improvment over old
make. It's enough to see the progress of the make, while not
obscuring the warnings. This by itself should help in cleaning up
the tree.

Note to anyone who wants to try this themselves: with the kbuild 2.5
patches applied, nothing changes (and the old build system is used)
unless you add '-f Makefile-2.5' to the make command. It does not
appear to be necessary to supply a bzImage target, and in fact,
Makefile-2.5 doesn't recognize it. That's basically all you have to
know.

Detailed Timings
----------------

Old kbuild, out of the box build:

424.18user 29.65system 7:07.94elapsed 106%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (1552796major+1856222minor)pagefaults 0swaps

touch fs/ext2/inode.c incremental build:

24.08user 2.04system 0:25.94elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (114294major+80954minor)pagefaults 0swaps

kbuild 2.5, out of the box build:

341.29user 18.71system 5:54.87elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (1080599major+1427293minor)pagefaults 0swaps

touch fs/ext2/inode.c incremental build:

13.75user 1.76system 0:15.31elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (122776major+59338minor)pagefaults 0swaps

--
Daniel


The email above turned into a lengthy thread on the lkml. It starts here and here.

Paul Komkoff's KBuild 2.5 patch for the 2.4 kernel follows:


From: Paul P Komkoff Jr
To: Linux Kernel Mailing List
Subject: [ANNOUNCE] kbuild 2.5 ports for -pre9 and -pre9-ac1
Date: Thu, 30 May 2002 23:55:44 +0400

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

I'm here to announce updates of kbuild 2.5 patch to apply on 2.4.19-pre9 and
2.4.19-pre9-ac1

Remember - I'm not Keith Owens - I'm just impressed by his work.
There is no non-i386 portions in my patches (yet) but they will added (I
believe).

Comments, patches, and other stuff are welcome.

You can grab patches here:
http://stingr.net/l/kbuild25-for-2.4.19-pre9.bz2
http://stingr.net/l/kbuild25-for-2.4.19-pre9-ac1.bz2

or
ftp://stingr.net/pub/l/kbuild25-for-2.4.19-pre9.bz2
ftp://stingr.net/pub/l/kbuild25-for-2.4.19-pre9-ac1.bz2

I'm also thrown away IKCONFIG feature in -ac, but It will be reenabled in
future versions

- --
Paul P 'Stingray' Komkoff 'Greatest' Jr /// (icq)23200764 /// (http)stingr.net
When you're invisible, the only one really watching you is you (my keychain)

-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAjz2g50ACgkQyMW8naS07KTLjQCgpgGEp0P1lT9OzWBHYeWlthvL
xBkAn3vB1v7uwPSpBTrYUZ9vYIRVDEZl
=zEFT
-----END PGP SIGNATURE-----

From: Paul P Komkoff Jr
To: Linux Kernel Mailing List
Subject: [ANNOUNCE] 2.4.19-pre9-ac1-s1
Date: Fri, 31 May 2002 00:59:29 +0400

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Here goes 2.4.19-pre9-ac1-s1.

I am have kbuild-2.5 in here too. So make -f Makefile-2.5 :)

Changes from kbuildified 2.4.19-pre9-ac1:
* Netfilter hooks for ethernet bridging (Lennert)
* MTU problem fixes for eepro100, tulip, (?)
and 3c59x
* Zerocopy path for netfilter (?)
* Alternate conntrack hash function & (me)
hash table sizing policy
* task_t.state cleanups (me)
* /dev/epoll (Davide Libenzi)
* Reiserfs quotas (?)
* smbfs large file support (Urban Widmark)
* irqrate (Ingo)
* Some hashes (yes they shouldn't be (wli)
here)
* Some reiserfs fixes (Chris Mason & namesys)
* bkl_rollup (Dave Hansen)
* e100 & e1000 2.4 backports (Intel, J.A.Magallon)
* bridging code fixes (Lennert)
* EVMS 1.0.1 (EVMS team)
with crap needed for O(1) sched (me)
* fast_walkA3-2.4.19-pre8 merge (Hanna Linder, Al Viro)
* Huge collection of netfilter P-O-M (Netfilter team)
patches applied with small quickfixes
from mine
* -aa bits, some of them merged to (Andrea Archangeli)
nothing, some remains
* kbuild 2.5 makefiles for all above (me)

Full, unformatted bk changelog available here:
http://stingr.net/l/ChangeLog-bk-2.4.19-pre9-ac1-s1.bz2
ftp://stingr.net/pub/l/ChangeLog-bk-2.4.19-pre9-ac1-s1.bz2

It should apply on top of plain 2.4.19-pre9-ac1. You can grab it here:
http://stingr.net/l/patch-2.4.19-pre9-ac1-s1.bz2
ftp://stingr.net/pub/l/patch-2.4.19-pre9-ac1-s1.bz2

Remember - you have to get patches to samba to make it actually work
with large files.

Enjoy.
- --
Paul P 'Stingray' Komkoff 'Greatest' Jr /// (icq)23200764 /// (http)stingr.net
When you're invisible, the only one really watching you is you (my keychain)

-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAjz2kpQACgkQyMW8naS07KR3uQCeIdQdRRUNC6T9xfMRXNfpZdQ9
OcsAoIzfbA67aJUI0+LlBZOj2k515HOV
=KMvE
-----END PGP SIGNATURE-----

Linus won't take this into the kernel...

Anonymous
on
May 31, 2002 - 11:27am

... until Keith Owens can hack it so doesn't come with source code, and costs money for commercial use, and sends data about what you do back to headquarters.

Linus

Anonymous
on
May 31, 2002 - 11:24pm

If you are reading this Linus, STOP BEING A FUCKING ASSHOLE AND TELL KIETH EXACTLY WHAT YOU WANT FROM HIM.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.