In my workplace, I use a customized version of Novell's ZENworks imaging

boot CD, which is based off of Linux. I have one particular model of

laptop - the IBM/Lenovo R61 - on which three different things fail

completely in current kernels (tested with 2.6.24.2 and 2.6.25-rc1):

USB, AHCI (and thus access to the SATA drive), and networking. As a

consequence of all three failing in parallel, I have no practical way to

get logs and other information off of the machine to help with tracking

down the bugs.
I am primarily concerned about the AHCI and networking issues, since

they are what need to be working in order for us to do what we need to

with these boot discs and these laptops. However, I intend to focus on

the USB issue first, because it seems slightly more tractable and fixing

it would allow me to reliably get logs off of the computer so as to

provide information to help track down and fix the other problems.
Specifically, the USB issue is more tractable in that I have found one

narrow set of circumstances in which I *can* get it to work, and so have

been able to obtain an lspci log and a dmesg log from the failing

laptop. I seem to remember the lkml FAQ advising not to simply attach

such files unsolicited, so I have not provided them here, but I am more

than willing to send them (and the matching .config file) along upon

request. Instead, I will do my best to summarize the errors as I have

observed them, though that best may be somewhat poor. In the following,

unless explicitly specified, I am using 2.6.25-rc1, simply because I

expect that it will be more likely to get attention and fixes than

earlier (released) versions.
Early in the boot process, immediately after the 'io scheduler foobar

registered' lines, the message
====

0000:00:1a.7 EHCI: BIOS handoff failed (BIOS bug?) 01010001

====
appears twice. Despite the parenthetical suggestion, I do not believe

that the problem could be a bug in the BIOS, because Windows is able to

access all of the hardware on these la...
[ message continues ]

add CC (Andrew, Greg and linux-usb)
--

Thanks,

Oliver

--

...
To make a long story short, the USB symptoms you describe indicate a

problem with interrupt routing.  This could well explain the other

difficulties too.  There are various kernel parameters you can try

putting on the boot command line to work around it:  acpi=noirq or

acpi=off or pci=noacpi or a few others.
Alan Stern
--

(Note: I consider it blatantly incorrect to send a reply both to a

mailing list and directly to the address of someone who is subscribed to

that list unless you have reason to believe that that someone will not

see the message otherwise, but in this case I am doing so anyway because

I see no way to avoid it and still make sure all relevant people receive

the message.)
I have now tried all three of these, with no apparent effect; the USB

drive is still not detected when plugged in after boot. A naive search

on Google provides no indication of other possible parameters to try;

the only list I have found of ACPI-related kernel parameters includes no

others which seem likely to be helpful without more knowledge of the

specifics of the situation (and the subsystem) than I have.
What would the next step be?
--

    Andrew Buehler

--

Regardless of how you consider it, that is how responding to these big

lists -must- work.
There is no practical way for respondents to know, without spending at

a minimum several minutes of their time per reply, whether or not the

explicit receipients of a message are or are not also on one or more of

the receiving lists.
Do you really expect, Andrew, that I should examine the membership lists

of each of linux-scsi, linux-usb and linux-kernel (if they are even open

to the public) to see if you're subscribed to them, before responding to

a message addressed such as this?
As subscribers and submitters to such lists, we just have to learn to

deal with this reality.  For example, I receive an average of a 100

messages per hour on this email address, -after- my employers spam

filters have knocked off over 90% of the incoming.
May I recommend you become an expert in procmail?  That or speed

reading (and speed ignoring ;).
This is not a matter of taste on these big lists.  There is no other

practical alternative.  Most of the burden of ultimate filtering must

be shifted to the recipients, and the senders asked only that they

err on the side of including every individual list or person already

on the address lists.
I'd recommend fastmail.fm as one of the least broken, most tech savvy

mail services.  I believe that their free side includes IMAP, though

not POP support.
--

                  I won't rest till it's the best ...

                  Programmer, Linux Scalability

                  Paul Jackson <pj@sgi.com> 1.940.382.4214

--

(Since there are multiple Andrews on just the LKML, and at least two -

one of whom is much more prominent than I am - in the direct address

list for this discussion, I'm not sure whether or not this is a
As I have now acknowledged twice (and this makes three times), there

does not seem to be a practical way to avoid it in this instance. That

does not make it any less incorrect to send a duplicate private copy to
AFAIRK (though I could be mistaken), procmail is not available under

Windows, which is what I have to use for work purposes. I have an

interest in learning it form my own purposes, but it is very much on the
I'm not as fond of IMAP as I used to be, though I no longer remember

exactly why, but I thank you for the recommendation. When I have

opportunity I will check it out, though that will probably not be this

week. (I also thank Joseph for the confirmation that the problem does

lie with Gmail.)
And, since there is no longer anything specifically kernel-related in

this subthread, I do not intend to reply publicly in it again unless

requested to do so.
--

    Andrew Buehler

--

aha -- my condolences ;)
Take care.  Your last reply made as much sense

as we're likely to make of this one.  Thanks.
--

                  I won't rest till it's the best ...

                  Programmer, Linux Scalability

                  Paul Jackson <pj@sgi.com> 1.940.382.4214

--

I (and a lot of other people as well, to judge by the email I receive)

don't think this is incorrect.  For one thing, it's not always possible

to tell whether or not the recipient is subscribed to any of the lists.

For another, getting two copies of a message is no big deal -- more

irritating (IMO) is getting a rejection message as a result of replying

to message which was cross-posted to a closed list.  But in each case,

hitting the "d" key will delete the unwanted message.
In fact, the thing that bothers me the most is when people reply to a

long email with just a few lines of new text but don't bother to prune

the long message down to its essential parts.  This forces me to read

through hundreds of lines containing nothing new or of interest in 
People on LKML who are more familiar with interrupt routing problems

might be able to offer more help.  For now, you can try things like

turning on CONFIG_USB_DEBUG, posting the output from dmesg, posting the

contents of /proc/interrupts (say before and after a new USB device is

plugged in).
Assuming that the 2.6.23 kernel works on your computer, you can go the

extreme route of installing git and doing a bisection to find the first

patch causing your difficulty.
Alan Stern
--

(I did not expect to receive any significant response to this part of

the message, and any discussion ensuing from it is unquestionably

offtopic for the kernel mailing lists. I will respond here for the

moment rather than sending a separate, private mail, but I am more than

willing to take further responses on this topic offlist or drop the
This is indeed why I did not see any way to avoid it in this instance.

However, that has no bearing on the question of whether or not it is
I disagree.
(For that matter, I am not in fact getting two copies; in fact, I am not

receiving through the list either messages which I send to the list or

messages which are sent both to the list and to me. Whether this is the

list's fault or Gmail's I don't know, though I suspect the former - but

it is very irritating, because it means that I do not see this thread on

the mailing list itself and have to carry on list-related discussion
I keep complete archives of everything I receive except for spam and (in

some cases) duplicates. Deleting received mail is a highly unpalatable

option, and in any case the ability to delete it is not the point; the

important question is whether it should have been there to begin with.
When I receive a message sent directly to me in a discussion which is on

a list, I expect that it is because someone either considered it

important enough to warrant making certain it came to my attention

specifically, or wanted to continue the discussion but felt that it
On the matter of quoting correctly and snipping correctly, you are

preaching to the choir where I am concerned. My standards for how much

context to leave in before beginning to snip are perhaps a bit looser

than those of some people (I leave up to three levels of quote in total

unless more is strictly necessary, since I find that fewer than that

frequently does not provide enough context if one does not have the

discussion in recent memory), but in principle I agree with you
In my current testing kernel, which I ...
[ message continues ]

This dmesg shows that ACPI is not enabled in your kernel config - most

likely this is the problem.  Try to enable it:
 1) In the "Power management options" submenu enable the "Power

    Management support" option (CONFIG_PM) - if this option is

    disabled, you will not see the option to enable ACPI below.
 2) In the same submenu enable the "ACPI (Advanced Configuration and

    Power Interface) Support" option (CONFIG_ACPI).
Without ACPI support the kernel can use legacy interrupt routing

tables from BIOS, but on new systems these tables are often broken due

to lack of testing (because all modern operating systems use ACPI

instead of these legacy tables).

Everyone has his own taste.  Obviously there's no world-wide consensus,

possibly because different people have different workflow habits and so
Sometimes that is the case but often it isn't.  Your expectations are

at variance with other people's behavior; you shouldn't expect everyone

else to change just to match your personal ideals.
On the other hand, I would be perfectly happy to edit your name out of

the reply list -- but since you said you aren't receiving all the

messages in this thread via the list that might not be a good thing to 
is an indication that interrupt routing is indeed not working right.

Or possibly your EHCI controller isn't working.  You could try

blacklisting or unloading ehci-hcd to see if that helps.  Of course 
Notice that the interrupt count for IRQ 11 doesn't change when you plug

in the device.  Obviously something is wrong there.
In fact, it's a little surprising that almost all the USB controllers

are routed to the same IRQ.  However this is beyond my area of

expertise.  You could try posting a message on the linux-acpi mailing 
There are some excellent tutorials on the web, with detailed

explanations of how to do a bisection to track down a kernel bug.
Alan Stern
--

I am well aware that this particular point is opinion. I have had

justifications for and arguments in favor of it in the past, but none of

them come readily to mind at the moment, except for the one gone over
Messages sent to my address directly are explicitly not filtered into

the folders I have set up for various mailing lists, so that if someone

does send me a "heads up" reply for a specific topic on a list to which

I am subscribed it does not get caught by the list filter and fail to

come to my attention. If a message fails to be filtered into any

mailing-list folder, then I should be able to conclude that it is

specifically intended for me, and not part of normal mailing-list

traffic. The practice of sending replies to both addresses renders this

an invalid conclusion. I do not think that it is unreasonable to expect
It's not that I'm not receiving all of this thread's messages via the

list - it's that I'm not receiving *any* of them via the list, and I

suspect that the reason is that my address is in both the To:/Cc: and

the list itself. Something is filtering it such that I do not receive

"duplicate" replies in this way, but it is doing so by filtering out the

list copy rather than the direct copy. I have seen mailing lists which

do this before, but I see no other indication that the LKML is one of

them, and I would not be in the least surprised if this turned out to be

yet one more problem with gmail.
As far as I am aware, I am seeing all messages posted to the list which

do not have me in To: or Cc:. I suspect that if a reply in this thread

were posted to the list but not sent to me, I would see it on the list.

It might be worth an experiment, but since it would increase traffic for
ehci-hcd is not modular in my current kernel, and if there is a way to

turn it off without its being modular I am not aware of it. I will have

to jump through a few hoops to be able to obtain a copy of the boot CD

with an updated kernel while not at work, but I will try to do so

sometime...
[ message continues ]

It is; Gmail will refuse to POP more than one copy of a mail to you,

no matter how many copies it recieves via different paths.  Which copy

you get seems to be dependant on which arrives first, so you can't

even hope a mail exchange will consistently arrive in one mailbox or

another.
Note that this also applies to mails cross-posted to multiple lists

you maybe be suscribed to; this breaks threading fantastically.
Google is aware of the issue, and considers it a feature.  If you find

another free mail service which isn't so broken, I'd love to hear

about it.
---
That said, netiquette on the kernel lists is to *never* drop CC's.

Too much traffic crosses the lists for anyone to read it all and note

anything they might be interested and/or implicated in.  Never

dropping CC's allows busy people to keep track of conversations

they've taken part in or that someone thinks they should see

without the worry of missing any important parts of one.
Or at least it does if your mail system isn't broken.  We get what we

pay for.  :-/
--

Joseph Fannin

jfannin@gmail.com

--

It's not unreasonable.  Neither is Aristotelian physics.  Nevertheless,

neither one is a good match to reality.
Why not arrange instead that messages sent from a mailing list server

_do_ get filtered into the corresponding folder, even if they were also

sent to your address?  This certainly should make your assumption (that

messages not filtered into any mailing-list folder are specifically
Well, I _am_ receiving your messages by way of linux-usb as well as
Go into the /sys/bus/pci/drivers/ehci_hcd directory.  Then for each

symbolic link to a controller device listed there, write the device's

name (with "echo -n") to the "unbind" file.  For example,
	echo -n 0000:00:1d.7 >unbind
ACPI isn't directly related to USB; rather it has to do with

transferring information between the OS and the

BIOS/vendor-specific-hardware.  Power management is example where such

a transfer is needed.  In your case, the relevant information is which

IRQ is connected to which motherboard device.  If you don't have ACPI

enabled in your configuration, then perhaps that's the problem -- try
Maybe they aren't.  But when you have multiple bugs, you have to fix
I don't know, but you should wait until the simpler problem is sorted

out before tackling the more complicated ones.
Alan Stern
--

Apparently it was the problem; enabling ACPI has fixed not only the USB

problem but also the network problem (somewhat miraculously, since I'm

quite certain that I had ACPI enabled in a 2.6.23.x kernel where the

network did not work despite an apparently matching driver).
I feel somewhat foolish for having reported a regression over what turns

out to have been a simple misconfiguration, but I still do think it's

somewhat misleading at best for something so potentially important to

completely non-power-related things to be buried under the heading of

power management... I would suggest moving it somewhere else in the

config and the dependencies, except that I have neither a suggestion for

a possible place nor any idea of how much actual work that would

involve.
With those two problems out of the way, what is left is the hard-drive

issue, and that is also halfway fixed by enabling ACPI. Specifically, it

is "fixed" in that the kernel sees the hard drive and I can mount it,

but it is not fixed in that the program we need to use in this

environment does not see the drive.
I have a config from a boot disc running 2.6.5 (that's not a typo) under

which the program in question *does* see the drive, but there are

massive differences between that config and the one I am using now, and

narrowing the critical difference down is likely to be somewhat

difficult - particularly since some of the "differences" are merely

renamed config symbols (i.e. the CONFIG_SCSI_SATA_*->CONFIG_SATA_*

switchover), and I have limited ability to tell which without intensive

investigation. Are there any established techniques for simplifying this

kind of comparison?
--

    Andrew Buehler

--

The only established technique is to run various kernels intermediate

between the one that works and the one that fails.
Alan Stern
--

(I suspect that some of the existing CC:s can now be dropped, and others

might need to be added if indeed this is worth discussing on kernel

lists at all, but I don't know what the protocol on that is so I have

left all of them in for the moment.)
Its detect-hardware-and-report mode shows a HD size of 0 (which is what

it has showed in cases where the kernel has not detected the drive), its

detect-partitions-and-report mode shows no partitions and no devices

which can have partitions, and attempting to actually use it to pull

down a drive image (it's a disk-imaging program) causes it to hang at

the point where it would begin to write.
Hmm. One thing which just sprang to mind, in the stab-in-the-dark

category: in 2.6.24.2, launching the program on some machines gave

warnings along the lines of "this program is using a deprecated ioctl,

please convert it to SG_IO" (which I naturally cannot do since it's

closed and I don't have the source), but IIRC in the 2.5.25-rc2-based

disc with ACPI enabled no such message appears. If the reason that there

are no longer such messages is that the ioctl in question has now been

removed, that might explain why the program does not see the drive.
I have suspected that I might eventually need to port the old interface

forwards to be able to continue to use this program, but I did not
I'm not sure I expressed myself clearly. I do not think the problem is

with the different kernels. I think the problem is with the different

configurations. I am asking if there are any established techniques for

comparing differences between config files from widely different

kernels.
Or, if you're suggesting running various kernels with configs which are

hybrids of the config which works and the current one which does not: in

order to do that I have to be able to tell what the actual differences

are, and at minimum the renamed symbols (not all of which I expect to be

able to identify at a glance) would make that quite difficult to do, so

I remain with the same ...
[ message continues ]

Could be.  You can use strace to find out what system calls the program 
Not as far as I know.
Alan Stern
--

It's provided by Novell, with whom I have no direct contact and am not

presently authorized to speak on behalf of my organization. From what I

have read about the history of their support on this program and these

discs, I do not expect that they would be willing to support it except

in environments which they provide in monolithic form; it would be

possible for me to copy an updated version of the program out of such an

environment to use in my own customized one, but I am not certain that

they have even created such an updated version, and in any case

obtaining it would almost certainly require buying the latest version of

Novell ZENworks - which my organization is certainly not prepared to do

at the present time.
In other words: I don't think that's likely to be practical in the

present instance. If you have reason to believe otherwise (past positive
Oh, well... thanks anyway.
Is there any place (aside from maybe the kernel changelog, which

contains a whole lot - if not several lots - of unrelated information)

where I could find a list of config-symbol name additions, changes,

deletions and meaning changes by version or by date? That would at least

let me build a mapping between the symbols in the older config and the

ones in the new one, which is about where I would have to start.
--

    Andrew Buehler

--

Not as far as I know.
Alan Stern
--

I know he's in the CC:, but I'm not sure he's reading this thread, and

I'm hesitant to bother people about things out of the blue unless I have

reason to expect that it's something they're going to care about. (Just

because it's a big deal for me doesn't mean it makes one whit of

difference to anyone else, and from what little I've seen on
Then I suppose I'm reduced to browsing the Kconfig files, reading old

changelogs, and trying a lot of different configs... thanks anyway.
--

    Andrew Buehler

--

In this case you shouldn't worry about it.  I don't know whether Greg

has been following this thread either, but I do know that he spends a

lot of time and effort trying to improve vendors' support for Linux.

This is right up his alley.  What I'm not sure about is the extent of

his influence over Novell...
Alan Stern
--

Heh, yes, I've been reading this.
It sounds like an old version of a Novell product is making a newer

kernel spit out a warning message.  Odds are this is fixed in a newer

one, as well as the basic issue that Novell doesn't even ship anything

based on the 2.6.24 kernel yet, so perhaps the Zenworks developers don't

even know of the issue, that's what bugzilla.novell.com is for :)
thanks,
greg k-h

--

Originally that was all that it was, but now I am seeing the product in

question not even see the hard drive despite the fact that it is

mountable with standard utilities. Whether the failure is in the program

or elsewhere I don't know, but I'm hoping it's in the program, because

if it's elsewhere I have a *lot* of tedious digging and testing ahead of
I haven't been able to find a newer version of the program, and do not

have the standing with my organization to contact Novell on their

behalf. (When I brought the idea up with the people who do have such

standing, in another context, the answer I received was essentially

"wait until we upgrade the servers to that version, which will not be

soon".) I also have not been able to find a clear contact path in any
The problem has been present at least since 2.6.23.x and I think since

2.6.18, but from what I've been able to find Novell's last kernel was

based on 2.6.16.
I didn't even know there *was* a bugzilla.novell.com - and I've been

digging around Novell's site (and Googling around it from outside) for

what seems to me like quite a while. From the looks of things, however,

they don't have a category for ZENworks or for imaging, and the Linux

I'm using is not one of the SUSE-based distros they do list. (The

environment on the boot disc itself doesn't really qualify as any kind

of distro at all.)
Unfortunately, from what I've seen elsewhere it looks as if they are A:

not interested in supporting use any Linux except for their own SUSE for

this purpose and B: not likely to be open to supporting or even helping

with a customized environment; all of their technical documentation on

the subject is geared towards adding files and drivers to their existing

environment, not to replacing the entire kernel of that environment or

working with the results. Given that I'm building complete replacement

kernels and customizing rather a number of other things in the end

product, I'm rather inclined to suspect that even if I got into contact

...
[ message continues ]

You could start by running the program under strace.  That should give

you a good idea of where the problem begins.
Alan Stern
--

Two reasons that I can think of off the top of my head.
One of them I mentioned above: because that precludes the possibility of

someone sending me a direct copy to draw my attention to something which

they think needs it, unless they send it separately from the list copy.

(This does not especially apply on the kernel-related mailing lists,

since no one is likely to think I am particularly worth drawing in to

any discussion there anytime soon, but it has come up elsewhere and the

basic principle is the same.)
The other is that this would lead to duplicate copies of the same reply

in the mailing list folder, which is ugly at best, especially with
It is indeed not enabled, and when I check the config for the 2.6.23.1

kernel where USB works, I find that it is enabled there. I will test the

result of enabling it in the current kernel. If I don't have an answer

by the end of the day, I probably won't be able to get one until at
Oh, I agree - that is a large part of why I posted a "full" description

of only one problem initially, rather than all three in a single mail.
--

    Andrew Buehler

--