From: Ingo Molnar <mingo@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 3:27 pm 2007

* Boaz Harrosh  wrote:
> > Honestly, the driver is probably rarely run by people that lack the

> > hardware, I bet...

>

> It was all "flight by instruments only". I called for HW testers and

> none came forward. All these changes, apart from "successful downgrade

> to messy" where also needed in order to push important changes to

> scsi.

>

> But a little bird said that QEMU might simulate this HW. SO I guess it

> is QEMU time for me.
heh. Incidentally i was thinking about using KVM for automated testing.

Important pieces of hardware should get an in-KVM simulator/emulator,

that way developers who do not own that hardware can do functionality

testing too. So basically the highest-quality drivers would have an

"inverse driver" in KVM, which simulates the hardware. (that model is

evidently useful to the hardware maker even for new hardware: it can

then also be used to test the Linux compatibility and Linux performance

of future planned releases of the hardware, etc.)
	Ingo

-



From: Linus Torvalds <torvalds@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 3:38 pm 2007

On Mon, 15 Oct 2007, Ingo Molnar wrote:

>

> heh. Incidentally i was thinking about using KVM for automated testing. 
Using emulators to test device drivers is almost certain to be pointless.
The problem with device drivers tends to be timing issues, odd hardware

interactions, and lots of strange (and sometimes undocumented) behaviour

and dependencies (eg things like "you have to wait 50us after setting the

reset bit until the hardware has actually reset").
These are all things that you'd generally not catch in emulation - because

the emulation by necessity is only going to be a very weak picture of the

real thing.
So I suspect you can find the easy stuff, but only by writing insanely

complex device model descriptions in the emulator environment itself, and

only for those device models that have actually been written.
Can it be donein theory? Sure. Practically? Not so sure. Would it help? I

suspect the effort to do the device model would be many times bigger than

*any* conceivable effort to just fix the driver bugs as they get found

through other means.
So we could perhaps have *really* good emulated hardware for a few models

of hw out there, but likely they'd be fewer and less varied platforms than

most kernel developers end up having hidden under their desk anyway.. 
			Linus

-



From: Alan Cox <alan@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 4:08 pm 2007

On Mon, 15 Oct 2007 12:38:06 -0700 (PDT)

Linus Torvalds  wrote:
>

>

> On Mon, 15 Oct 2007, Ingo Molnar wrote:

> >

> > heh. Incidentally i was thinking about using KVM for automated testing.

>

> Using emulators to test device drivers is almost certain to be pointless.
For some things. I do it a bit because you can use it to fake

failures that are tricky to do in the real world. It won't tell you the

driver works but its suprisingly good for testing for races (forcing IRQ

delivery at specific points), buggy hardware you don't posess, and things

like media failures and timeouts your real hardware refuses to do.
Alan

-



From: Linus Torvalds <torvalds@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 4:15 pm 2007

On Mon, 15 Oct 2007, Alan Cox wrote:

>

> For some things. I do it a bit because you can use it to fake

> failures that are tricky to do in the real world. It won't tell you the

> driver works but its suprisingly good for testing for races (forcing IRQ

> delivery at specific points), buggy hardware you don't posess, and things

> like media failures and timeouts your real hardware refuses to do.
Heh. I do agree that you likely find bugs, even if quite often it's

exactly because the behaviour is something that will never happen on real

hardware.
But failure testing is very useful - I forget who it was who debugged some

driver by taking a CD and just scrathing it mercilessly to induce read

errors ;)
Having a really *bad* HW emulator can certainly work that way too, even if

it also would probably end up hitting just a few of the potential error

paths..
			Linus

-

From: Ingo Molnar <mingo@...>
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 6:23 pm 2007

* Linus Torvalds  wrote:
> On Mon, 15 Oct 2007, Ingo Molnar wrote:

> >

> > heh. Incidentally i was thinking about using KVM for automated

> > testing.

>

> Using emulators to test device drivers is almost certain to be

> pointless.
something like that wont enable 100% coverage (or even reasonable

coverage for most hardware), so it's no replacement for actual hard

testing, but it could push out the domain of minimally tested code quite

a bit and increase the quality of the kernel. Races are always tough and

so are bugs on the side of the hardware, but it's the silly boot-time

crash showstoppers and "device does not work anymore" mistakes that

causes us to lose most of the testers and early adopters.
I'm not really worried about the 1% of bugs that are tough to find and

fix (because they are actually fun to find and fix), i'm worried about

the 99% easy and boring bugs - because they annoy users just as much as

the exciting bugs do. If we fix them faster then there's more time (and

more tester stamina) left for the harder to find bugs.
so i think adding redundancy in form of a simplified hw emulator can

certainly not hurt and fundamentally increases robustness - and it will

definitely reduce the chance for a whole host of stupid bugs that are

not in the hardware but are in the ~4 million lines of Linux driver

codebase. Limits and scalability would also be testable: "if i put 32 of

these networking cards into a Linux box, will the Linux driver blow up".
not that i think this is realistic for any significant portion of the

hardware currently - unless some hw maker starts doing it. But KVM will

have a good portion of the core PC platform emulated (APIC, etc.) - and

that's a nice step forward already.
	Ingo

-