Speaking of file sizes, xz[1] already provides better compression:

xz -k -9 linux-2.6.34-rc2.tar

     55320408 linux-2.6.34-rc2.tar.xz

xz -e -k -9 linux-2.6.34-rc2.tar

     54800808 linux-2.6.34-rc2.tar.xz

One drawback of xz is that it's not multi-threaded, much like bzip2 or 
gzip; would be great if it could be changed.

[1] http://tukaani.org/xz/


-- 
Tomasz Chmielewski
http://wpkg.org

--

[ message continues ]

For some time there has been a multi-threaded bzip2 called 
pbzip2[1], for some time; hell even Debian has it :)

I have no idea why the original poster is trying to say how "all teh 
awesome" his code is being faster, well 'duh' it is using all the cores 
on $BOX rather than just a single one.

I would be interested in comparisons against pbzip2 and the amusingly 
named pigz[2]...plus a bunch of memory use comparisons, my AR7 board 
only has 16MB of RAM :)

Cheers

[1] http://compression.ca/pbzip2/ - supports stdio (de)compression
[2] http://www.zlib.net/pigz/ - no idea if this supports stdio

-- 
Alexander Clouter
.sigmonster says: Approved for veterans.

--

[ message continues ]

I asked explicitly to be CC'd. I saw somewhere that more than 200 messages 
are posted to lkml each single day, so excuse me for not subscribing to 
it. Of course nobody needs to give a damn about what I ask for, but then 
it's difficult for me to respond.

You didn't quote where I said 'how "all teh awesome" my code is being 
faster'.

Anyway, please download a bz2 kernel tarball, and decompress it with 
pbzip2 and then with lbzip2. Eg. on a quad-core AMD or similar. Then 
please re-read the last sentence of section (2) of my previous mail.

Or please check my site which I linked to previously, and look at some 
test reports and their analysis on the debian-mentors mailing list (link 
on my site, likewise).

I could go on about how the multiple workers decompressor of lbzip2 is 
designed, or about its signal handling, but since you decided upfront that 
I'm an attention whoring idiot, I won't struggle. I'll paste at the end 
one such test report I generated now on a 2x2 core Opteron 275, with the 
test file being "linux-2.6.33.1.tar".

I'd call lbzip2 useful especially if kernel.org continued to offer the 
full tarballs in the current .bz2 format. ("I dare to recommend lbzip2 in 
order to shorten both compression and decompression times for whomever 

lbzip2's malloc() peaks were like this during the correctness (not the 
performance) test phase -- not that you'd care:

- single-threadded decompression of bzip2-compressed file:  9,955,696
- same but with four worker threads:                       71,308,096
- single-threaded decompression of lbzip2-compressed file:  9,955,696
- same but with four worker threads:                       65,079,712
- compression with four worker threads:                    49,049,265

(lbzip2 has a compile-time configurable "backlog factor", sort of the 
maximum number of buffered input chunks per thread. It is 4 per default 
which I deem an okay tradeoff between memory usage and presumable, 
unexpected bursts in (de)compression ...
[ message continues ]