From: Alain Knaff <alain@knaff.lu>

This is an updated version of my bzip2/lzma patch

It is based on an idea by Christian Ludwig, includes support for
compressing the kernel with bzip2 or lzma rather than gzip. Both
compressors give smaller sizes than gzip.  Lzma's decompresses faster
than bzip2.

It also supports ramdisks and initramfs' compressed using these two
compressors.

The functionality has been successfully used for a couple of years by
the udpcast project

This version applies to "tip" kernel 2.6.27-rc5

Changes since last version (update2):

- Removed NEW_CODE #ifdef : non-Intel architectures will be patched
  eventually anyways
- Replaced IN_MEMORY #ifdefs by (less efficient) runtime checks

Signed-off-by: Alain Knaff <alain@knaff.lu>

---

diff -uNrp -X linux.trees.git.udpcast/Documentation/dontdiff linux.trees.git/arch/x86/boot/compressed/head_32.S linux.trees.git.udpcast/arch/x86/boot/compressed/head_32.S
--- linux.trees.git/arch/x86/boot/compressed/head_32.S	2008-09-07 11:29:12.000000000 +0200
+++ linux.trees.git.udpcast/arch/x86/boot/compressed/head_32.S	2008-09-09 00:41:04.000000000 +0200
@@ -143,7 +143,7 @@ relocated:
 	pushl %eax	# input_len
 	leal input_data(%ebx), %eax
 	pushl %eax	# input_data
-	leal boot_heap(%ebx), %eax
+	leal _end(%ebx), %eax
 	pushl %eax	# heap area
 	pushl %esi	# real mode pointer
 	call decompress_kernel
@@ -184,8 +184,6 @@ relocated:
 .bss
 /* Stack and heap for uncompression */
 .balign 4
-boot_heap:
-	.fill BOOT_HEAP_SIZE, 1, 0
 boot_stack:
 	.fill BOOT_STACK_SIZE, 1, 0
 boot_stack_end:
diff -uNrp -X linux.trees.git.udpcast/Documentation/dontdiff linux.trees.git/arch/x86/boot/compressed/head_64.S linux.trees.git.udpcast/arch/x86/boot/compressed/head_64.S
--- linux.trees.git/arch/x86/boot/compressed/head_64.S	2008-09-07 11:29:12.000000000 +0200
+++ linux.trees.git.udpcast/arch/x86/boot/compressed/head_64.S	2008-09-09 00:41:09.000000000 +0200
@@ -287,7 +287,7 @@ relocated:
  */
 	pushq	%rsi			# Save the real ...
[ message continues ]

^^^^

There is no reason to include header files conditionally.

It might be worthwhile to create a proper subdirectory rather than using 
an underscored namespace (also, in general, dashes are preferred in 


Should this be always_inline or just plain inline?

	-hpa
--

[ message continues ]

I hate to contradict you, but in that case, lots of other filenames
include/linux/8250_pci.h             include/linux/linux_logo.h
include/linux/ac97_codec.h           include/linux/lm_interface.h
include/linux/acpi_pmtmr.h           include/linux/memory_hotplug.h
include/linux/adfs_fs.h              include/linux/minix_fs.h
include/linux/adfs_fs_i.h            include/linux/mm_inline.h
include/linux/adfs_fs_sb.h           include/linux/mm_types.h
include/linux/affs_hardblocks.h      include/linux/mmu_notifier.h
include/linux/agp_backend.h          include/linux/mnt_namespace.h
include/linux/aio_abi.h              include/linux/mod_devicetable.h
include/linux/anon_inodes.h          include/linux/msdos_fs.h
....

    282     282    7598

Any other people want to chime in about this before I make update5? I'm
all for adopting common coding standards, but only if they are actually

As you may have understood, the decompressors themselves are samples
and/or reference implementations downloaded from elsewhere, and thus
might not be up to the coding standards of the rest of the kernel.

... but the same can be said of inflate.c (the existing gzip
decompressor), which would not pass muster either (and indeed, the only
scripts/checkpatch.pl WARNINGS of my patch are about funny indentation
of inflate.c that happen in the context of my changes...)

There's a fine line to walk here between "bringing up" these codes to
standard (at the risk of breaking them...) or touching the decompressing
engines as little as possible.


Same remark: this is 3rd party code which I preferred to touch as little
as possible...

Alain
--

[ message continues ]

Not arguing that.  However, in this case I think the right thing is to 

I'm familiar with that.  However, I think in this case it is preferrable 
to use the all-kernel facilities, since there is an obvious direct mapping.

	-hpa
--

[ message continues ]

What do you think about update5? Any things still to improve, or is it
already scheduled for inclusion in one of the upcoming kernels? If so,
which one?

Thanks for any info,

Alain

--

[ message continues ]

I was out of town for LKS/LPC last week, and I didn't get a chance to 
deal with it before I left.  However, it's definitely getting there.

I will add it to the -tip tree, and assuming it doesn't cause massive 
breakage it probably will be pushed into .28.

	-hpa
--

[ message continues ]

Thanks,

Alain
--

[ message continues ]

Just did so, and it broke my first test build.

arch/x86/boot/compressed/head_64.o: In function `relocated':
(.text+0x28): undefined reference to `_end'
make[3]: *** [arch/x86/boot/compressed/vmlinux] Error 1
make[2]: *** [arch/x86/boot/compressed/vmlinux] Error 2
make[1]: *** [bzImage] Error 2
make: *** [sub-make] Error 2

	-hpa
--

[ message continues ]

Just leave the changes in head_64.S away; this was only about improved
memory management. It is not strictly needed, as the change in
/include/asm-x86/boot.h still makes sure that the heap is big enough,
even for bzip2

Regards,

Alain

--

[ message continues ]

OK, this kind of underscores something I have been thinking already: 
this is a really large patch and should be broken down into a patchset. 
  Presumably something like:

- Cleanups to inflate library code
- Use existing library code for initramfs
- Add bzip2 library code
- Add lzma library code
- Add support for bzip2/lzma initrd
- Add support for bzip2/lzma bzImage

	-hpa
--

[ message continues ]

More failures...

/bin/sh: scripts/bin_size: No such file or directory

Happens with O=.

	-hpa
--

[ message continues ]

Could be a permission issue. Did you try to chmod 755 scripts/bin_size ?
Hmmm, diff does indeed not encode permissions, how should such cases be


Alain
--

[ message continues ]

I mean I'm using O= to build in a different object directory.  It 
sometimes means one needs $(src) in places.

	-hpa
--

[ message continues ]

Sorry, $(srctree), not $(src).

	-hpa
--

[ message continues ]

Could you please post an example of such a command line?

Thanks

Alain
--

[ message continues ]

[Trimming Cc: list]


mkdir o.junk
make O=o.junk defconfig
make O=o.junk

	-hpa
--

[ message continues ]

Thanks. That's interesting, especially when testing stuff for several
architectures.

Alain

--

[ message continues ]

Hi Alain,

Are you planning to submit an updated patch any time soon?  If so, 
please separate the ARM, x86, library and generic portions into separate 
patches.  It looks like at least some of them already went into ARM, 
which makes it impractical to include this as a monolithic patch, which 
it really shouldn't have to be, anyway.

	-hpa
--

[ message continues ]

I'll look into it (the split) this weekend, if I'll find the time.
Should each part be compilable on its own? If so, it might be difficult
to do the split along the lines outlined above.

Alain

--

[ message continues ]

Not individually, but part 1 should compile, as should parts 1+2, etc.

This pretty much means the order should be:

1. add library functions
2. generic functionality
3. x86 functionality
4. ARM functionality

Soem of these may be obsolete; I noticed collisions with the ARM tree.

	-hpa
--

[ message continues ]

Unfortunately, due to the nature of the patch, it will be hard to
separate out "x86 functionality" from changes in lib/inflate.c . Indeed,
a large part of the patch consists in moving some gzip-specific headers
and internal variable declarations from the callers:
arch/x86/boot/compressed/misc.c on one hand, and init/do_mounts_rd.c and
init/initramfs.c on the other hand into lib/inflate.c

So, leaving out the x86-specific change
(arch/x86/boot/compressed/misc.c) in the first change, would force to
leave that change out of lib/inflate.c as well (or else, the
above-listed items would be doubly defined). But, if I left out these
changes of lib/inflate.c, I'd need to leave them out of and
init/do_mounts_rd.c and init/initramfs.c too (or else the above-listed
items would not be defined at all in that situation). Can you suggest a
solution? I could theoretically break that dependency chain using an
#ifdef (as was the case until patch 3), but apparently #ifdef's are
highly frowned upon. Or was it just the name of the ifdef ("NEW_CODE")
that you objected to? Another option would be to (temporarily) keep 2
copies of lib/inflate.c around, but somehow that doesn't feel right.


Great! Could you tell me where to download the ARM tree from, so that I
can have a look?

Thanks,

Alain

--

[ message continues ]

Temporarily leaving two copies would work, as would be leaving in an 
#ifdef as long as it is gone by the end of the patch series.

	-hpa
--

[ message continues ]

Anybody can help here?

Thanks

Alain
--

[ message continues ]

A few more thoughts:

quiet_cmd_bzip2 = BZIP2    $@
cmd_bzip2 = (bzip2 -9 < $< ; $(size_append) $<) > $@

quiet_cmd_lzma = LZMA    $@
cmd_lzma = (lzma -9 -c $< ; $(size_append) $<) >$@

These will produce stale files on failure.  Using stdout in Makefiles is 
a bit tricky.  You probably need something like:

(bzip2 -9 < $<  && $(size_append) $<) > $@ || (rm -f $@ ; false)

	-hpa
--

[ message continues ]

either you find a way to chmod +x from a makefile (but this is a bit tricky)
or better, you call it with $(CONFIG_SHELL) :

   $(CONFIG_SHELL) scripts/bin_size

Some other commands in the makefiles are already handled this way.

Willy

--

[ message continues ]