Rusty Russell's new module loader was recently merged into Linus' 2.5 kernel tree [story]. This new implementation aims to cleanup and reduce the amount of code in the kernel and user space required to load a kernel module. Additionally, it now removes the requirement that kernel and user space code for modutils have to be in sync.
A recent exchange on the lkml between Keith Owens [interview] and John Levon [interview] brought to light debugging issues that have been introduced with this new code. Keith points out that as there is no longer any section information available, debugging with ksymoops and kgdb has now become quite difficult, if not impossible. Keith says, "The complete lack of kernel and module symbols (no /proc/ksyms) means that ksymoops is now useless on 2.5 kernels. If you get an oops in a kernel built without kksymoops, there is no way to decode the oops."
I spoke with Keith to further understand these problems, and he was kind enough to offer a full explanation. Read on to learn what has changed, and how it affects module debugging.
Keith Owens: Rusty claims that it reduces both the amount of kernel code and user space code required to load a module. I am not convinced that this claim is true, because the new loader is still missing features of the old loader.
Rusty's other claim is that it breaks the restriction that kernel and user space code for modutils have to be in sync. This is true, it has been difficult to add new module features while still keeping backwards compatibility. However the restriction could have been removed without moving everything into the kernel, it just needed a new module syscall that was extensible.
JA: With the new code, is it now possible to safely unload a kernel module?
Keith Owens: No, all the races still exist. The new code attempts to reduce the races and introduces new interfaces for module writers. Only experience will show if the new interfaces are any better at closing the races or if we still have unload problems.
JA: How big of an effort is it going to be to add back the section information necessary for debugging and profiling modules?
Keith Owens: The kernel has to publish where each section is loaded for each module, the size of each section, where the module was loaded from and the timestamp of the module that was loaded. IOW, all the information that used to be in the __insmod_* entries in /proc/ksyms.
Also the new module loader has to save this data whenever it changes. Without a snapshot of the module state after each change, you cannot decode an oops correctly, especially if the oops has killed the machine.
Note that kksymoops is not the complete answer, because it is optional and increases kernel size by ~ 15%. Adding another 15% of unpageable storage to the kernel just to get back the debugging features is not a good idea.
JA: Does this affect just the debugging of modules, or the rest of the kernel as well?
Keith Owens: The kernel can be debugged without this data. However it would be a lot easier if both the kernel and modules were displayed using the same set of data. It is always easier to use one source of data than two, especially when they have completely different formats.
From: Keith Owens
To: linux-kernel mailing list
Subject: New module loader makes kernel debugging much harder
Date: Sat, 23 Nov 2002 13:20:10 +1100
The new module loader makes kernel debugging much harder.
There is no section information available, which means ...
* ksymoops cannot decode oops in modules. Without section data, there
is no way to reliably determine where module symbols were loaded.
* kgdb cannot debug modules. gdb needs to know which modules are
loaded and where each section of each module starts.
* kdb cannot display the section for a symbol. Which means the user
cannot do
objdump -j --adjust-vma= module
to map the code to the original object and source.
The complete lack of kernel and module symbols (no /proc/ksyms) means
that ksymoops is now useless on 2.5 kernels. If you get an oops in a
kernel built without kksymoops, there is no way to decode the oops.
Big step backwards, Rusty.
From: John Levon
Subject: Re: New module loader makes kernel debugging much harder
Date: Sat, 23 Nov 2002 02:35:13 +0000
On Sat, Nov 23, 2002 at 01:20:10PM +1100, Keith Owens wrote:
> The complete lack of kernel and module symbols (no /proc/ksyms) means
> that ksymoops is now useless on 2.5 kernels. If you get an oops in a
> kernel built without kksymoops, there is no way to decode the oops.
Additionally, module profiling is not possible any more.
> Big step backwards, Rusty.
Somebody (this includes Rusty himself) needs to come up with a workable
solution. For my own needs, the start address of the mapped module is
good enough, but it seems you need more than that. I'd be quite happy
with the re-introduction of /proc/ksyms as it would mean no userspace
code changes for me :)
regards
john
--
Khendon's Law: If the same point is made twice by the same person,
the thread is over.
From: Keith Owens
Subject: Re: New module loader makes kernel debugging much harder
Date: Sat, 23 Nov 2002 13:43:20 +1100
On Sat, 23 Nov 2002 02:35:13 +0000, John Levon wrote:
>On Sat, Nov 23, 2002 at 01:20:10PM +1100, Keith Owens wrote:
>
>> The complete lack of kernel and module symbols (no /proc/ksyms) means
>> that ksymoops is now useless on 2.5 kernels. If you get an oops in a
>> kernel built without kksymoops, there is no way to decode the oops.
>
>Additionally, module profiling is not possible any more.
>
>> Big step backwards, Rusty.
>
>Somebody (this includes Rusty himself) needs to come up with a workable
>solution. For my own needs, the start address of the mapped module is
>good enough [snip]
Only if you assume that the .text is at a known offset from the start
of the module. There are multiple programs that need to know where
each section really is, instead of making assumptions about how a
module is laid out.
From: John Levon
Subject: Re: New module loader makes kernel debugging much harder
Date: Sat, 23 Nov 2002 16:23:46 +0000
On Sat, Nov 23, 2002 at 01:43:20PM +1100, Keith Owens wrote:
> Only if you assume that the .text is at a known offset from the start
> of the module. There are multiple programs that need to know where
> each section really is, instead of making assumptions about how a
> module is laid out.
Yes, sorry, that is what I meant.
regards
john
--
Khendon's Law: If the same point is made twice by the same person,
the thread is over.
From: Keith Owens
Subject: Re: New module loader makes kernel debugging much harder
Date: Sun, 24 Nov 2002 11:49:32 +1100
On Sat, 23 Nov 2002 16:23:46 +0000, John Levon wrote:
>On Sat, Nov 23, 2002 at 01:43:20PM +1100, Keith Owens wrote:
>
>> Only if you assume that the .text is at a known offset from the start
>> of the module. There are multiple programs that need to know where
>> each section really is, instead of making assumptions about how a
>> module is laid out.
>
>Yes, sorry, that is what I meant.
Even for module profiling, we need section data available.
Although all module code currently goes in a single text area, there is
no guarantee that will always be the case. In the past we have had
multiple text areas in modules (out of line lock code used its own
section for a long time) and future changes could require multiple text
sections. To do profiling correctly, you need to know where all the
text sections are, i.e. the module loader has to publish symbols and
section data. Loosing that data is a big step backwards.
From: John Levon
Subject: Re: New module loader makes kernel debugging much harder
Date: Sun, 24 Nov 2002 01:06:17 +0000
On Sun, Nov 24, 2002 at 11:49:32AM +1100, Keith Owens wrote:
> Even for module profiling, we need section data available.
I know, that's why I agreed with you...
> Although all module code currently goes in a single text area, there is
> no guarantee that will always be the case. In the past we have had
> multiple text areas in modules (out of line lock code used its own
> section for a long time) and future changes could require multiple text
> sections. To do profiling correctly, you need to know where all the
> text sections are, i.e. the module loader has to publish symbols and
> section data. Loosing that data is a big step backwards.
Actually all *I* need is the address of one of the sections, as long as
they're simply mapped in we can work backwards given the sample file
and the offset of that section. So this already worked for when the
locking code was a separate section. But yeah, it would be helpful for
other uses to have more info available.
regards
john
--
Khendon's Law: If the same point is made twice by the same person,
the thread is over.
From: Keith Owens
Subject: Re: New module loader makes kernel debugging much harder
Date: Sun, 24 Nov 2002 12:18:46 +1100
On Sun, 24 Nov 2002 01:06:17 +0000, John Levon wrote:
>On Sun, Nov 24, 2002 at 11:49:32AM +1100, Keith Owens wrote:
>> Although all module code currently goes in a single text area, there is
>> no guarantee that will always be the case. In the past we have had
>> multiple text areas in modules (out of line lock code used its own
>> section for a long time) and future changes could require multiple text
>> sections. To do profiling correctly, you need to know where all the
>> text sections are, i.e. the module loader has to publish symbols and
>> section data. Loosing that data is a big step backwards.
>
>Actually all *I* need is the address of one of the sections, as long as
>they're simply mapped in we can work backwards given the sample file
^^^^^^^^^^^^^
>and the offset of that section. So this already worked for when the
>locking code was a separate section. But yeah, it would be helpful for
>other uses to have more info available.
How do you know if the sections are "simply mapped"? The module loader
could assign different sections to different mappings, there is no
guarantee that they are contiguous. They were contiguous using the 2.4
module loader but only because the syscall only allowed for a single
area. The new loader can assign sections anywhere it likes.
One possibility the new loader opens up is the ability to replicate the
pure module data (rodata and text) for each node of a NUMA box. There
is already an option to replicate the kernel text on each node to cut
down inter-node traffic. Replicating the pure module data would be
nice as well. I guarantee that will result in something that is not
"simply mapped".
From: John Levon
Subject: Re: New module loader makes kernel debugging much harder
Date: Sun, 24 Nov 2002 01:34:51 +0000
On Sun, Nov 24, 2002 at 12:18:46PM +1100, Keith Owens wrote:
> How do you know if the sections are "simply mapped"? The module loader
> could assign different sections to different mappings, there is no
> guarantee that they are contiguous. They were contiguous using the 2.4
> module loader but only because the syscall only allowed for a single
> area. The new loader can assign sections anywhere it likes.
Mmmm, good point. So I will need full section info too. And some
userspace changes to cope with such ...
thanks
john
--
Khendon's Law: If the same point is made twice by the same person,
the thread is over.
From: Rusty Russell
Subject: Re: New module loader makes kernel debugging much harder
Date: Mon, 25 Nov 2002 12:26:58 +1100
On Sun, 24 Nov 2002 12:18:46 +1100 Keith Owens wrote:
> One possibility the new loader opens up is the ability to replicate the
> pure module data (rodata and text) for each node of a NUMA box. There
> is already an option to replicate the kernel text on each node to cut
> down inter-node traffic. Replicating the pure module data would be
> nice as well. I guarantee that will result in something that is not
> "simply mapped".
Ewww, you are a sick lad 8)
But I'm not sure that there is any interface which wouldn't break horribly
when faced with that possibility.
The patch to restore /proc/ksyms is trivial. As is the addition of a start
entry /proc/modules. When combined with rth's simplified loader, it should
be sufficient for both ksymoops and oprofile.
kgdb needs a patch to work, anyway: you might want to restore /proc/ksyms
in that patch? (I don't use kgdb, so my ignorance here is complete).
Rusty.
--
there are those who do and those who hang on and you don't see too
many doers quoting their contemporaries. -- Larry McVoy
Rusty?
It's a pity Rusty has not said anything with regard to this yet; I'd be quite curious to hear what he has to say.
Also, here's some food for thought: as valid as Keith's points are, shouldn't he have raised them before the new module loader was actually merged, while it was still being discussed on lkml?
--
schnee
It was raised
Several people raised concerns about the new module loader. I (Keith
Owens) told Rusty in no uncertain terms that he was destroying the
debugging information.
Rusty knows...
I spoke to Rusty offlist. He seems to be set on
keeping the simply-mapped property at the moment.
rth's new loader removes the separate allocation of
init/core areas. I think it's likely to end up as
patch to /proc/modules.
I agree that kksymoops is not ideal, but ...
[btw, I can no longer log in to this site, and if I
get a new password mailed it *still* won't work. Jeremy ?]
re: password
Drop me an email at jeremy@kerneltrap.org and we'll get it figured out. I'll need to know what browser you're using, what's your login ID, what happens when you try to login, etc...