Hi,

a topic that just came up in kernel summit is a script to remove
unneeded kernel configs automatically to reduce the compile time.
Incidentally, I already wrote such a script during the last SUSE hack
week a couple of weeks ago, so I'd like to share here, hopefully to
give an idea for further improvements.

The script checks the currently loaded modules and trims other
CONFIG_XXX=m simply, and tries make oldconfig, and writes out the
resultant .config in the current directory after some checks. 
You can specify the config file via option,  as default, it reads from
/proc/config.gz. 

The script is VERY hackish.  I should have begun with perl or whatever
better script language, but I chose bash and co.  So, don't expect
much code quality.  I'm no script guy after all :)


Takashi
---

#!/bin/sh

# this magic makes grep and co a lot faster
export LANG=C

kroot=
subarch=$(uname -i)
oldconfig=/proc/config.gz
localversion=
modulelist=
optimize_smp=
optimize_memory=
outputfile=

usage () {
    cat <<EOF
diet-kconfig -- Trim down kernel configs of unused modules
Usage: diet-kconfig [-options]
   Create a kernel config file based on the running system
option:
  -r ROOT  Specifies the linux-kernel root, default: current dir
  -a ARCH  Architecture (e.g. i386), default: running arch
  -c FILE  Old config file, default: /proc/config.gz)
  -o FILE  Output config file, default: kernel/.config
  -l LIST  Module list, default: read lsmod
  -v VER   Specifies the local version string, default: as is the original
  -s       Optimize SMP
  -m       Optimize memory model (for i386)
  -h       This help
EOF
    exit 1
}

while getopts r:a:c:o:l:v:pmh opt; do
    case "$opt" in
	r)
	    kroot="$OPTARG"
	    ;;
	a)
	    subarch="$OPTARG"
	    ;;
	c)
	    oldconfig="$OPTARG"
	    if [ ! -f "$oldconfig" ]; then
		echo "Cannot find old config $oldconfig"
		exit 1
	    fi
	    ;;
	o)
	    outputfile="$OPTARG"
	    ;;
	l)
	    modulelist="$OPTARG"
	    if [ ...
[ message continues ]

Check my AutoKernConf (http://cateee.net/autokernconf/), it has
huge (and automatically generated) database about drivers, modules,
and configuration, and it gives a minimal kernel configuration for the
most of the hardware hardware.

It needs a real maintainer and tools for non-hardware drivers
(fs, net, ...) and special hardware (CPU capabilities,

-i is wrong (gives "unkwnown" on Debian).


My python code works better (handle "alias", etc.).
Anyway I should send some cleanup patches to correct and
improve some ugly kernel Makefile.


ciao
	cate
--

[ message continues ]

At Thu, 18 Sep 2008 16:01:50 +0200,

Thanks, this looks good.  Nice to see other people already working


OK, such distro specific things have to be handled better.  I didn't
know of /boot/config-* at all.

OTOH, if /proc/config.gz exists, it's the right place to see because
of an obvious reason.   So, checking other paths should be a kind of

We don't need to handle alias because the whole module names
come from lsmod output.  That's so simple.
OTOH, I don't think the parsing in my script is perfect at all,
though.  It's a result of one or two hours hack.
Improving that function a bit more robust would be pretty helpful.

Regarding the kernel config database: it's nice, indeed.
But, I don't think the whole database is required for the task I tried
to achieve.  It wasn't intended to create a kernel config from
scratch.  It's not necessarily the absolute minimum.  Instead, it just
trims the unnecessary module builds.  For that purpose, the database
is not needed by assumption of the all required modules have been
loaded.  This is rather a safer option than creating a minimum kconfig
from scratch.


thanks,

Takashi
--

[ message continues ]

Hi Takashi,

On Wed, 17 Sep 2008 01:55:26 +0200

I agree with Giacomo: you should also check for /boot/config-`uname -r`
and /boot/config-`uname -r`.gz.

Also, if the idea is to make life easier for kernel newbies, I think the better
would be to have the script called at kernel Makefile (something like "make
diet"), and having a few other config's available somewhere at kernel tree,
since even a "minimal" kernel may need (or not) a few random modules, like
usb-storage. So, I think we should open a dialog that allows the selection of
using the previous kernel config or selecting between a few profiles like
"minimalist" (just the auto-detected things), "desktop" (adding things like
usb-storage), "notebook" (with a power-saving optimized config),
"server" (adding stuff like LVM, RAID5 and some advanced network configs).

It seems a good idea to check at /proc/cpuinfo and optimize for that
specific processor, enabling SMP only if needed, and only for the number of

Using just a shellscript and binutils seems to be better than using other
tools, since it allows the usage on minimal configured systems where the user
might not have perl or other scripting languages.

Cheers,
Mauro.
--

[ message continues ]

But very slow, complex and not enough powerful:
The tools should have extensive knowledge about Kconfig,
and the CONFIG_ dependencies; it should handle a lot of
structures (text/binary; line/record oriented; etc.), which
it is not easy in shell (awk, od, sed with extended regexp are
not core utilities).

So my suggestion is: do it in a high level language, to find and
stabilize the design, and then integrate it in kconfig later (using
C).

ciao
	cate

--

[ message continues ]

I don't think that performance is a big issue here. I tested Takashi 
scripts on this old 1.5 GHz centrino one core notebook, and it runned in a 
few seconds. Kernel compilation with diet config took 30-40 mins. The 
additional time for the script didn't significantly increased the total 


Since CONFIG_ and their dependencies are constantly 
changing/adding/removing at kernel, the better would be to have a good 
Kconfig parsing, avoiding to have lots of "local knowledge" about symbols. 
The better model seems to be the one where subsystem maintainers don't 
need to take extra care for not breaking the tool when touching at 
Kconfig/Makefiles.

Also, it occured me that some drivers have "implicit" Kconfig dependencies 
inside that may limit its work. Looking on a random driver, just as an 
example, you have things like:

$ grep ifdef cpufreq_stats.c
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS

So, I suspect that the script might also need to parse the selected files 
to seek for additional rules.

I'm not sure what to do after discovering such things: Assume the option 
that generates smaller code? Assume the option that generates the most 
complete code? Maybe the better would be to use the selected config 


Seems a good approach to me.

-- 
Cheers,
Mauro
--

[ message continues ]

Instead of replying all other emails, it is simpler to explain my
approach.

The purpose of my program is to find all CONFIG_s, which are needed to 
support all the hardware (and some more), and to show them in 
menuconfig. This is the main difference with Takashi purpose:
he wants to build a bootable (without initramfs) kernel, starting
to an high modular kernel (usually found in distribution),
so no usable on an already optimized machine.

Anyway there are a lot of overlapping tasks, which we could
exchange.

My design (files and sample output are in 
http://cateee.net/sources/autokernconf/autokernconf-2008-08-29/ ):

kdetect.sh: detects hardwares, in a redundant manner, in order
to have data if there are missing one ore more between: /proc,
/sys, lspci, ...

autokernconf.sh: a shell script which read the driver database
(lkddb.list) and kdetect output; it generates the desired CONFIG_s.

So far, so simple, and similar to Takashi method. I only split
the task in two programs (detection and configuration).

The big difference is in the additional program, which generated the
driver database: it reads all sources, it checks 25 subsystems
(see http://cateee.net/lkddb/statistics.html ), taking in account
to preprocessors (#ifdef CONFIG_...), #defines, static strings, masks, 
all architectures, etc.
Additionally it parse Makefile and Kconfig to find the dependencies
of every source file.

It misses some generic detection (CPU, memory), which Takashi is
doing, and the kconfig integration.

So I think most part could be integrated with the Takashi proposal, and 
doing optionally the parsing of full driver database.

ciao
	cate




--

[ message continues ]

At Fri, 19 Sep 2008 11:12:34 +0200,

Yes.  Your stuff would be actually helpful for people who want

Right, for example, kconfig parser can be shared.

Agreed.

I think a good start would be to integrate a simple version of parser
and trim-down mechanism into the upstream kbuild system, so that you
can call such as "make diet", as Mauro mentioned.

BTW, Steve Rostedt told in the last night that he uses also his own
stuff for the same diet kconfig. 
Steve, can we share your script, too? 


thanks,

Takashi
--

[ message continues ]

Incidentally, I wrote such a script as well. In 64 lines of python code.
I attach the script in question in the mail. I use a approach very
similar to yours. But I work around some of the nasty part.

It needs to start with kernel config has most of the modules enabled.
Most of the kernel config from the linux distribution is like that.
Then it try to parse the Kbuild looking for what kernel config produce
what module. It is not a 1 to 1 mapping.

Once  we have the mapping. It open /proc/modules to go over every
module. Remove the config which can produce those module from the
mapping.

Now we have a blacklist of config produce modules but not the one we want.

Then  the last stage is just open a config file and filter out the config
option on the blacklist. Write the result to a new config file.

I avoid arch/ and firmware/ because it is nasty. I figure module in
arch/ is small enough I don't mind building it. And the firmware directory,
if I don't build the module loads it. Those firmware will automatically skipped
any way.

The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)
        obj-m += cmd640.o
endif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"

For easier parsing.
It is pretty fast as well. Less then a second if the file cache is
hot.

$ time minmod.py .config
reading Kbuild Makefile 593 files
acpi_cpufreq : Uknown module, may be firmware?
dcdbas : Uknown module, may be firmware?
1560 module option disabled
Writing new config to: .config.min

real	0m0.470s
user	0m0.303s
sys	0m0.075s
$

Chris
--

[ message continues ]

Oops, I forget to attach the script. Here it is.

Chris
[ message continues ]

At Fri, 19 Sep 2008 16:55:51 -0700,


I think a whitelist would be useful, too, e.g. for hotplug devices


This should be fixed in Makefile.
Care to submit a patch?


Takashi
--

[ message continues ]

Are you sure?
I.e. if you set CONFIG_BLK_DEV_CMD640=y, you see that
the two constructs gives different results.

If the makefile author did such complex construct, I
really think there is a reason ;-)

ciao

--

[ message continues ]

Yes indeed, we should find out what the author meant.

 ifeq ($(CONFIG_BLK_DEV_CMD640), m)
	obj-m += cmd640.o
 endif

and

 obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o

mean two different things.

The former means to compile the cmd640 only if it is set to a module.
The later means to compile the cmd640 if it is either a module or built 
into the core kernel code.

The author needs to make this change or explain why it is like this.

-- Steve

--

[ message continues ]

At Mon, 22 Sep 2008 11:41:36 -0400 (EDT),

FYI, in this particular case, cmd640.o is linked in the upper Makefile

Right.  My guess is the link order, but still there must be a better
way to fix.


thanks,

Takashi
--

[ message continues ]

The white list is marginally useful. But it is very complicate to do.
In order to preform white list. You have to know the dependency
of the module. e.g. You need to know exactly which options to
turn on so the modules will show up. The current dependency
analyze is the other way around. It knows which device should
show up when some config option is enabled. Answering the
question of which option should enable in order that device
show up is a much harder problem.

Also, as long as the user can get a boot able kernel out of
the minimal config. It is easy enough to recompile the kernel
with white list modules and install them as needed.

The white list should be a secondary step to do because
its complexity. The first priority is to have some thing simple
and provide a good minimal kernel module option. Then we
can go from there, how to add extra white list from it. The white
list is a very different game any way. I found the minimal config

Some module does not want to be compile as build-in.

Chris
--

[ message continues ]

At Mon, 22 Sep 2008 12:25:21 -0700,

That is true.  Enabling arbitrary kernel config item is hard to
achieve right now.  I think this feature should be implemented in
kbuild parser itself.  The current reverse-select is way limited and
known to be problematic in many kernel configs.

Anyway, I think the white list isn't that hard for certain use-cases.
My assumption is that users base on the distro kernel that have
already all modules.  Then, for likely scenarios such as USB-storage
hotplug, we can simply provide possible modules such as usb-storage
and nls_* as the additional modules.  The script would just need to
add them to the existing module list.  If necessary, the module

I don't think choosing necessary kernel config is "easy enough".  If
it were easy enough, we don't need such scripts at all.  The problem
is that you must know exactly which config should be used for what

I feel you goal is much farther than mine :)
The white list can be just a list of modules.  If it doesn't work,
that's fine -- at least, we get the exactly same system like currently

But not about this case.  It wants to be compiled-in, too.


thanks,

Takashi
--

[ message continues ]

I was looking at the kbuild system. BTW, I really like the Makefile
in kbuild.  For the reverse select, here is what I have in mind.

The reverse select needs to maintain the define-user chain for
each kernel option. And each kernel option has a list of the kernel
option and value pare to enable an option. Once we produce such
an list for "all config". We can know exactly what kernel option
needs to set in order to get to so module. It is kind of doing
the data flow analyze on the config options.


If you just want a white list without resolving the dependency, that
should be very easy to do in my current minmod.py.

In terms of implement in C. The currently kconf symbol has most
of the bits ready already. We can add a symbol type "module".
It points to a list of config symbols. That should be good enough.
Then we can use 1 bit of flag in config symbol to mark it is blacklisted
or not.

We can use some thing like minmod.py to provide a prototype
for some kernel user to try it out. If it covers most of the feature

I think my script can catch those already. Maybe I will beat it
up to generate a list of similar configs using "obj-m" directly.
We can example them one by one.

Chris
--

[ message continues ]

At Tue, 23 Sep 2008 11:15:19 -0700,

Well, it is not so easy, too.
The good-working reverse selection with dependency checks is a
feature we've long for in Kconfig, but it hasn't been implemented
because it could easily introduce a messy dependency loop, or some
cases are not easily resolvable.

Anyway, the development of reverse-selection should involve with Sam
and other Kbuild maintainer/developers.


Takashi
--

[ message continues ]

What I describe is a simpler version of that. The key difference is that,
I only work out the dependency base on a distro config file. So there
is only one set of config value to generate a module instead of many
combo in the more general case. I think that will cover 80 percent of
what user want. If you have a known all config file, the reverse selection

Care to CC them? I think most likely you need to touch the current
kconf for "make diet" any way.

Thanks

Chris
--

[ message continues ]

At Thu, 18 Sep 2008 17:43:48 +0200,


Maybe a good idea to add such functionality into kbuild itself.
It's THE parser after all.


thanks,

Takashi
--

[ message continues ]

At Thu, 18 Sep 2008 12:25:41 -0300,

OK, I'll add the check of these as fallback.

There are different perspectives for this.  My goal to reduce the
build time with the existing kernel config for a specific hardware.
That is, it makes eaiser to create another set of kernel with test
patches or bisect.

OTOH, it would be definitely useful if we can create a clean minimum
kernel config from scratch based on the hardware information.  For that,

Well, perl or python is most likely installed on every modern system,
so I think we don't stick with bash & co.  But, I see your point, and
it's also the reason I started with bash, etc.


thanks,

Takashi
--

[ message continues ]

One has to be careful here - I've more than once hit kernel bugs that I ended
up bisecting, which were dependent on the exact .config used.  The most recent
was a deadlock during early boot if CONFIG_HID_COMPAT=y, which would have
been turned off by any sane kconfig trimmer.  On the flip side, running a
trimmer wouldn't minimize my .config very much, because I've already set most
stuff I *never* use to 'n'.

And more than once, I've had a bisect go horribly awry and need to be re-started
because the bisect crossed back and forth over a commit that added a config
option, so doing a 'make oldconfig' at each iteration would re-prompt, and
I wasn't consistent in the reply I gave.  Usually doesn't matter, except when
the broken code is in the support for the feature...

I *do* keep around a *very* minimal "only what I need to get to single-user"
config if build time is an issue...

Might be different if you're starting off with a distro config that's basically
an 'allmodconfig' - lotta fat to trim off *that*....
[ message continues ]

At Thu, 18 Sep 2008 17:09:27 -0400,

That's true.  But, you should at least test the trimmed kernel once
before bisecting to reproduce the bug.
And, my script doesn't do anything so intrusive.  It trims just unused
CONFIG_*=m.  Since they are not loaded anyway, reducing them must be

Heh, you are obviously not a "typical" user ;)

My scenario was for users of distro kernel who want to help debugging

Bisect isn't perfect, indeed.  But, in many cases, it can catch
the harmful commit more easily, especially if it's in the area

Exactly.

thanks,

Takashi
--

[ message continues ]