After Cliff Wright posted a patch for vnd(4) to handle compressed disk 
images some time ago[1] and Florian Stoehr wrote a userland tool to 
(de)compress disk images (using the cloop2 format also found on other 
systems)[2], I sat down to play with the two things tonight.

In short: very successfully!


What I basically did was build a kernel and vnconfig(8) with the patch in 
[1], and build Florian's tool (which I received in private from him).
After booting the patched kernel, I did the following steps:

Verify I was running the kernel I just built (this is all in qemu):

 	qemubsd# uname -a
 	NetBSD qemubsd 3.99.6 NetBSD 3.99.6 (GENERIC) #1: Mon Jun 13 01:12:47 MEST 2005  feyrer@miyu:/usr/cvs/src-cloop2/sys/arch/i386/compile/obj.i386/GENERIC i386


Create some toy filesystem:

 	qemubsd# /root/vndz/vnconfig -l
 	vnd0: not in use
 	vnd1: not in use
 	vnd2: not in use
 	vnd3: not in use
 	qemubsd# makefs
 	usage: makefs [-t fs-type] [-o fs-options] [-d debug-mask] [-B endian]
 		[-S sector-size] [-M minimum-size] [-m maximum-size] [-s image-size]
 		[-b free-blocks] [-f free-files] [-F mtree-specfile] [-x]
 		[-N userdb-dir] image-file directory
 	qemubsd#
 	qemubsd# makefs inc.fs /usr/include
 	Calculated size of `inc.fs': 11157504 bytes, 1273 inodes
 	Extent size set to 8192
 	inc.fs: 10.6MB (21792 sectors) block size 8192, fragment size 1024
 		using 1 cylinder groups of 10.64MB, 1362 blks, 1408 inodes.
 	super-block backups (for fsck -b #) at:
 	 32,
 	Populating `inc.fs'
 	Image `inc.fs' complete


Mount the filesystem the usual (uncompressed) way:

 	qemubsd# /root/vndz/vnconfig vnd0 /tmp/inc.fs
 	qemubsd# /root/vndz/vnconfig vnd0 -l
 	vnd0: / (/dev/wd0a) inode 10839
 	vnd1: not in use
 	vnd2: not in use
 	vnd3: not in use
 	qemubsd#
 	qemubsd# disklabel vnd0a
 	disklabel: Invalid signature in mbr record 0
 	# /dev/rvnd0a:
 	type: vnd
 	disk: vnd
 	label: fictitious
 	flags:
 	bytes/sector: 512
 	sectors/track: 32
 	tracks/cylinder: 64
 	sector...
[ message continues ]

What does the "cl" stand for in the name "clcompress"?  I don't see any
obvious relationship between the name and what the tool does (which, if
I understand correctly, is to perform a special kind of compression on
files that will later be used by a modified version of vnconfig).  I
suggest using a name more like "vncompress".

--apb (Alan Barrett)

Right, we agreed on renaming it from clconfig to vncompress. :)


  - Hubert

Please call it "vndcompress" rather than "vncompress".  Actually,  

-- thorpej

I think the english word for the action being taken here is "compress", 
not "zip" (checked via http://dict.leo.org :), so let's go for 
"vndcompress" (and "vnduncompress").

Who's gonna rename vnconfig -> vndconfig?


  - Hubert

Hubert, you need "vnddecompress" since there is no English word uncompress.=
 :)

-Andrew

But NetBSD speaks unixese not english, and vndzip is a lot easier
	eh?  It sounds like english to me. (and it is listed on e.g. www.dictionary.com)

eric

-- thorpej

Shouldn't this just be a flag to vnconfig (which ought to be called
vndconfig, but never mind :)?

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

"cl" came from "cloop", that's what this is called on Linux.

-Florian

Note their is a bug when using OpenOffice on BSD read-only file system.
Linux seems to have the philosophy let it run no matter what, where as
BSD will report if it finds something wrong. This is seen in the use
of fnctl F_SETLK with OpenOffice in a read-only filesystem on a NetBSD
LiveCD. BSD says setting the lock does not make sense on a read-only
file system, and reports an error, where Linux just lets this go.
The quick hack patch I used is below, on file:
/usr/pkgsrc/misc/openoffice/work/store/source/fileunx.hxx

--- fileunx.hxx.orig 2003-08-15 04:53:11.000000000 -0700
+++ fileunx.hxx 2005-05-18 11:24:51.000000000 -0700
@@ -251,7 +251,7 @@
        lock.l_start  = 0;
        lock.l_len    = 0;
 
-       if (::fcntl (rhFile, F_SETLK, &lock) < 0)
+       if ((::fcntl (rhFile, F_SETLK, &lock) < 0) && errno != EINVAL)
        {
                // Save original result.
                storeError result;
Note the EINVAL could cover other cases, so I have not turned this in
to the OpenOffice people, as it may be to broad.
 Because this was a message response I posted here, but might be better
on tech-pkg.

On Mon, 13 Jun 2005 02:17:39 +0200 (CEST)

IMHO all the kernel changes need  to be conditionalized on some kernel
option. There are lots of vnd users out there, most of which never will use
this format.

Anyone ever looked at creating an uncompressing overlay filesystem, so that
you could store all files as bzip2 -9 encoded on an arbitrary filesystem
(e.g. iso9660), but without the .bz2 suffix, and read them uncompressed
throught the fs mounted on top of it?

Martin

Good point!


  - Hubert

Looking at the code, I wonder how to do this:
  * just add an #ifdef for a symbol around the code, and keep the
    data fields for compression handling in struct vnd_softc
  * put the #ifdef also on the fields in struct vnd_softc
  * use something else than an #ifdef/option (I have no idea on kernel
    hacking :)

Thinking more about the userland tool's name, I wonder if "vndcompress" 
would be more appropriate than cl<something>...


  - Hubert

Don't #ifdef the fields in the vnd_softc structure -- that would  

Yes, that sounds better.  Just mention which format it is compatible  
with.


-- thorpej

I recommend option 1. Unless the crypto fields are huge, leave them in. It=
=20
will make everyone's life easier.

Take care,

Bill

In message <Pine.LNX.4.61.0506131337200.31861@m24s24.vlinux.de>, Hubert Feyrer 

Unless the fields in vnd_softc are big, I'd include them 
unconditionally -- it simplifies life for anyone who wants to write a 
kernel data structure dumper.  (I'm thinking gdb rather than libkvm.)

		--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

Is there a good reason for leaving the data fields around?  (LKM
issues if the structure is shared with other parts of the kernel;


I apologize if I missed something earlier, but is this something that
could be folded into vnconfig ?

-allen

-- 
                  Use NetBSD!  http://www.NetBSD.org/

some vnd structures are shared with UVM and there is a vnd LKM that
i just noticed a few days ago i didn't enable in the build and will
be shortly.  is there really a good reason to make this code
conditional?  i don't see the point really...

I looked a bit, and it seems the vnd_softc(sp?) is only used inside the 
kernel, it's also protected by an appropriate #ifdef in the vndvar.h 
header. And as the vnd(4) driver is unlikely to interoperate with some 
other LKM (I hope?), I think it's safe to also include the data in the 


The tool compresses & uncompresses a disk-image (file), and isn't all that 
different from what compress/gzip etc. do - I don't feel like we should 
add this into vnconfig(8). Of course it could be done, but it would sound 
more like a hack to me (and the -c(ompress)/-u(ncompress) flags are 
already taken in vnconfig(8) ;-).

I'd leave it as seperate tool.


FWIW, the latest version of my patch is here (!!! beware, there are some 
bugs in there according to the author, Cliff Wright, which will raise a 
few problems !!!):

 	http://www.feyrer.de/Misc/vnd.gz-HF20050614.patch


  - Hubert

About the name...  I understand that you chose vncompress (without the
'd' in vnd) because it matches vnconfig.

However, vnconfig should be really named vndconfig, as the device is
vnd, not vn (this was discussed in the past, IIRC).  I'm wondering...
could we rename the utility (and provide a compatibility link for the
old name, of course)?

(Just asking this to avoid the addition of another tool with an
"incorrect" name.)

Cheers,

-- 
Julio M. Merino Vidal <jmmv84@gmail.com>
http://www.livejournal.com/users/jmmv/
The NetBSD Project - http://www.NetBSD.org/

I have no opinion on this. If it's decided that vnconfig gets renamed to 
vndconfig, I'd rather make the other tool named vndcompress/vnduncomopress 
then.

Maybe core or so wants to decide this. :)

  - Hubert

This is the usual aproach I'd recommend.

Martin

/usr/share/examples/mount_portal/advanced.1.conf contains something
which is a lot like what you want.

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert                            --  rhialto/at/falu.nl
\X/ Hi! I'm a signature virus! Copy me to your .signature to help me spread!

Heh, cool!

Martin

FWIW, Linux seems to use "create_compressed_fs" and 
"extract_compressed_fs" (see 
http://www.novell.com/products/linuxpackages/professional/cloop.html).
Not my first choice... cl2compress? clcompress? cl2zip? clzip?


  - Hubert

-- 
Still in exile.

Well, do we need to split this into two programs? Hm, don't think so (in 
my opinion).

Perhaps "clcompress" is suitable?

-Florian

Sure not two programs, at best we can make it regard argv[0].
What do you think about "vncompress", as this is really a vnd-thing, not a 
cloop (== linux' name) thing?


  - Hubert

Reading argv[0] is evil. Don't do that.

        - Nathan

> On Mon, 13 Jun 2005, Florian Stoehr wrote:
   > > Well, do we need to split this into two programs? Hm, don't think so
   > > (in my opinion).
   > >
   > > Perhaps "clcompress" is suitable?
   > 
   > Sure not two programs, at best we can make it regard argv[0].
   
   Reading argv[0] is evil. Don't do that.

it's already used all over our tree, including that we install only
one binary of grep instead of three for grep, egrep and fgrep.  i'm
not sure this argument is compelling for netbsd.


.mrg.

Why not?
It's what compress/uncompress and gzip/gunzip do, too...


  - Hubert

It may be historical practice, but it's bad historical practice. At
the very least you have to have a flag to set what to do, regardless
of the name. What does the program do when you run it as
"nbclcompress", or "clcompress-test", or whatever?

        - Nathan

Right now: complain & abort.
What do you suggest? Make only a "vncompress" that handles -c(compress) 
and -d(ecompress)/-u(ncompress)? That's what it did initially, but I 
really find using two names easier.


  - Hubert

I don't see anything wrong with having one program which accepts -c and
-d (or whatever options are picked) and then, for certain values of
basename(argv[0]), supplies one or the other by default.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Well, I guess it would break the principle of least surprise: if the default
is to compress, and you rename the decompressor to some other name, then you
won't be able to launch the newly created file with the same command line as
the former version (as it will no longer default to decompress but compress)

Regards.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)

iD8DBQFCrwUbMiWEUf7YMGERAm8VAKDbU0upBKOGEkhWfmelw2NNJ5jx0gCg2mMZ
o5qV/qHhhANgnEY3169dou8=
=M+WJ
-----END PGP SIGNATURE-----

> Cc: tech-kern@NetBSD.org


If.  No, I think the default should be more like...

 cflag =3D 0;
 dflag =3D 0;
 if (!strcmp(basename(av[0]),"vndzip")) cflag =3D 1;
 if (!strcmp(basename(av[0]),"vndunzip")) dflag =3D 1;
 ...process command-line flags...
 ... -c -> { cflag=3D1; dflag=3D0; } ...
 ... -d -> { dflag=3D1; cflag=3D0; } ...
 if (!cflag && !dflag)
  { fprintf(stderr,"%s: must give -c or -d when not run as `vndzip' or `v=
ndunzip'\n",__progname);
    exit(1);
  }

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Salut,


I'd suggest -d because that's what [bg]zip* use.

				Tonnerre

OK, that's a good name!

(Although, remember, we're 100% compatible, so our newly created files 
should also work on Linux right out of the box).

-Flo

That should go into the manpage. :)
BTW, what's status of this? ;)


  - Hubert

I'll fixup the manpage and send it to you as soon as we've finally chosen 
a name for the program.

-Florian