For the Debian installer we've been tracing a problem with our installation

boot floppy. If booted with 'expert' at the syslinux prompt, it only shows:

<snip>

SYSLINUX 3.70 <etc.>

boot:expert

Loading linux......................

Loading initrd.gz....ready.

Probing EDD (edd=off to disable)... ok

</snip>
And then the emulator crashes (both VirtualBox and qemu), or on real

hardware the system reboots. The qemu crash is included at the bottom

of this mail.
The problem was initially seen with the 2.6.25 Debian kernel and traced

to a set of Xen-related patches backported from upstream 2.6.26. Floppies

using Debian 2.6.24 or pristine 2.6.24/2.6.25 don't show the problem.
Bisection has shown the culprit to be this very early 2.6.26 commit:

$ git bisect bad

099e1377269a47ed30a00ee131001988e5bcaa9c is first bad commit

commit 099e1377269a47ed30a00ee131001988e5bcaa9c

Author: Ian Campbell <ijc@hellion.org.uk>

Date:   Wed Feb 13 20:54:58 2008 +0000
    x86: use ELF format in compressed images.
Important factor here is that we "pack" the kernel using upx [1] (in order

to fit everything on a floppy). The original (unpacked) kernel after this

commit boots fine, only a packed version fails.

We have tried upx versions 2.01, 3.01 and 3.03, all with same result.
Both "good" (before commit) and "bad" (after commit) images are available

at: http://people.debian.org/~fjp/tmp/d-i/floppy/upx/

Included are the boot floppy image, the raw kernel and the packed kernel.
The issue can also be reproduced using qemu without booting the floppy

itself. For the "bad" image:

# Boots correctly (but fails when mounting root fs):

$ qemu -kernel vmlinuz -hda /dev/zero

# Fails:

$ qemu -kernel vmlinuz.upx -hda /dev/zero
So, the primairy question here is:

- is this a kernel regression because whatever changed is no longer valid

  conform "kernel format specs", or

- is this a latent issue in upx that somehow creates an invalid image, or

- does this change effectively create a new "...
[ message continues ]

Ok, I do not consider this to be a regression. 
Clearly UPX is expecting a very specific kernel format, and clearly UPX

just needs to be updated for the changes.
That said, if some UPX person can explain what it is that UPX needs, maybe

we can add some format markers into the kernel (and perhaps some extra

header info) so that future format changes are transparent or at least

break in obvious ways.
			Linus

--

The issue with UPX is that UPX actually wants to replace the kernel

decompressor, which also does other tasks (relocation and ELF parsing)

in recent kernels.
	-hpa
--

It does save us about 5% though, which is quite substantial when you're

talking about fitting a kernel on a boot floppy. For the past few kernel

versions it has made the difference between being able to base

installation floppies on the standard Debian kernel (generic 486 flavor)

or having to come up with radical new solutions.
From my tests for this issue it looks like .26 has grown so much that

we've now reached that point anyway though :-(
Cheers,

FJP

--

Sounds to me like UPX makes assumptions about the kernel image format

that it really doesn't have standing to assume.
	-hpa

--

This is the first time I've looked at UPX but from glancing through the

code it certainly appears to make a lot of assumptions about the

structure of the bzImage (to the point that it looks for specific code

sequences within the binary).



It seems that the way UPX works is that it extracts the compressed image

from the bzImage, recompresses it and rebuilds a new bzImage replacing

the decompression stage (and possibly some of the other 16 bit startup,

I'm not quite sure yet) with its own. The issue is that its new

decompressor does not understand the ELF format and expects a raw

binary.
I think that UPX probably has gone a bit beyond the documented

interfaces, but it's not unreasonable way. In fact the changeset which

you referenced (or one of the ones around it) actually adds further

documentation (in Documentation/x86/i386/boot.txt) and header fields to

aid in doing the sort of extraction UPX wants to do and documents more

explicitly the formats which can be expected to be found there. Also

around the same time a checksum field was defined which is invalidated

by the repacking.



All in all I'd say it should be treated as a new subtype/variant which

UPX should support. I'd say adding support for bzImage v2.08 to UPX

would be pretty easy for someone who knows the code base (I couldn't

even find the decompressor code, but then it's pre-coffee time here..).
Ian.

--

Ian Campbell
Fame lost its appeal for me when I went into a public restroom and an

autograph seeker handed me a pen and paper under the stall door.

		-- Marlo Thomas
--

Thanks for the quick and complete reply Ian. I already expected this but

thought it would at least be useful to inform people here of the issue

(and get it indexed for search engines).
I have no problems with it not being treated as a bug/regression.
I've now filed a bug report against upx:

http://sourceforge.net/tracker/?func=detail&aid=2014835&group_id=2331&at...
Cheers,

FJP

--