Jaroslav Sykora posted a series of five patches to handle the kernel portion of what he described as "shadow directories [1]", providing an example which utilized FUSE to access the contents of a compressed file from the command line. His first example was cat hello.zip^/hello.c about which he explained, "the '^' is an escape character and it tells the computer to treat the file as a directory. The kernel patch implements only a redirection of the request to another directory('shadow directory') where a FUSE server must be mounted. The decompression of archives is entirely handled in the user space. More info can be found in the documentation patch in the series."
There were numerous problems suggested. Jan Engelhardt noted, "too bad, since ^ is a valid character in a *file*name. Everything is, with the exception of '\0' and '/'. At the end of the day, there are no control characters you could use." Later in the thread an lwn.net article [2] from a couple years ago was quoted, "another branch, led by Al Viro, worries about the locking considerations of this whole scheme. Linux, like most Unix systems, has never allowed hard links to directories for a number of reasons;" The article had been discussing Reiser4, which treats files as directories. In the current discussion, Al Viro added, "as for the posted patch, AFAICS it's FUBAR in handling of .. in such directories. Moreover, how are you going to keep that shadow tree in sync with the main one if somebody starts doing renames in the latter? Or mount --move, or..."
From: Jaroslav Sykora <jara@...>
Subject: [RFC PATCH 0/5] Shadow directories
[2]Date: Oct 18, 11:21 am 2007
Hello,
Let's say we have an archive file "hello.zip" with a hello world program source
code. We want to do this:
cat hello.zip^/hello.c
gcc hello.zip^/hello.c -o hello
etc..
The '^' is an escape character and it tells the computer to treat the file as a directory.
[Note: We can't do "cat hello.zip/hello.c" because of http://lwn.net/Articles/100148/ [3] ]
The kernel patch implements only a redirection of the request to another directory
("shadow directory") where a FUSE server must be mounted. The decompression of
archives is entirely handled in the user space. More info can be found in the documentation
patch in the series.
The shadow directories are used in RheaVFS project [ http://rheavfs.sourceforge.net/ [4] ],
and it also can be used with the original AVFS [ http://www.inf.bme.hu/~mszeredi/avfs/ [5] ].
The patches are against vanilla 2.6.23.
This is my first bigger contribution to the kernel so please be gentle ;-)
Jara
--
"Elves and Dragons!" I says to him. "Cabbages and potatoes are better
for you and me." -- J. R. R. Tolkien
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [6]
More majordomo info at http://vger.kernel.org/majordomo-info.html [7]
From: Jan Engelhardt <jengelh@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[7]Date: Oct 18, 12:05 pm 2007
On Oct 18 2007 17:21, Jaroslav Sykora wrote:
>Hello,
>
>Let's say we have an archive file "hello.zip" with a hello world program source
>code. We want to do this:
> cat hello.zip^/hello.c
> gcc hello.zip^/hello.c -o hello
> etc..
>
>The '^' is an escape character and it tells the computer to treat the file as a directory.
Too bad, since ^ is a valid character in a *file*name. Everything is, with
the exception of '[mid=347375,347391,347452,347602,347610,347887,347998]' and '/'. At the end of the day, there are no control
characters you could use.
But what you could do is: write a FUSE fs that mirrors the lower content
(lofs/fuseloop/however it was named) and expands .zip files as
directories are readdir'ed or the zip files stat'ed. That saves us
from cluttering up the Linux VFS with such stuff.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [8]
More majordomo info at http://vger.kernel.org/majordomo-info.html [9]
From: Jaroslav Sykora <jara@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[9]Date: Oct 18, 1:07 pm 2007
On Thursday 18 of October 2007, Jan Engelhardt wrote:
>
> On Oct 18 2007 17:21, Jaroslav Sykora wrote:
> >Hello,
> >
> >Let's say we have an archive file "hello.zip" with a hello world program source
> >code. We want to do this:
> > cat hello.zip^/hello.c
> > gcc hello.zip^/hello.c -o hello
> > etc..
> >
> >The '^' is an escape character and it tells the computer to treat the file as a directory.
>
> Too bad, since ^ is a valid character in a *file*name. Everything is, with
> the exception of '[mid=347375,347391,347452,347602,347610,347887,347998]' and '/'. At the end of the day, there are no control
> characters you could use.
>
> But what you could do is: write a FUSE fs that mirrors the lower content
> (lofs/fuseloop/however it was named) and expands .zip files as
> directories are readdir'ed or the zip files stat'ed. That saves us
> from cluttering up the Linux VFS with such stuff.
>
Yes, that's exactly what RheaVFS and AVFS do. Except that they both use an escape
character because:
1. without it some programs may break [ http://lwn.net/Articles/100148/ [10] ]
2. it's very useful to pass additional parameters after the escape char to the server.
We can start VFS servers (mentioned above) and chroot the whole user session into
the mount directory of the server. It works but it's very slow, practically unusable.
So both servers need some kind of VFS redirector. In the past there were many
different approaches -- LD_PRELOAD hack, CodaFS hack, NFS hack (?), proof-of-concept
kernel hacks (project podfuk) etc.
If anybody can think of any other solution of the "redirector problem", possibly
even non-kernel based one, let me know and I'd be glad :-)
--
I find television very educating. Every time somebody turns on the set,
I go into the other room and read a book.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [11]
More majordomo info at http://vger.kernel.org/majordomo-info.html [12]
From: David Newall <david@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[12]Date: Oct 18, 4:37 pm 2007
Jaroslav Sykora wrote:
> If anybody can think of any other solution of the "redirector problem", possibly
> even non-kernel based one, let me know and I'd be glad :-)
If I understand your problem, you wish to treat an archive file as if it
was a directory. Thus, in the ideal situation, you could do the following:
cat hello.zip/hello.c
gcc hello.zip/hello.c -o hello
etc..
Rather than complicate matters with a second tree, use FUSE with an
explicit directory. For example, ~/expand could be your shadow, thus to
compile hello.c from ~/hello.zip:
gcc ~/expand/hello.zip^/hello.c -o hello
I think no kernel change would be required.
I'm not keen on the caret. One of the early claims made in
http://lwn.net/Articles/100148/ [13] is:
> Another branch, led by Al Viro, worries about the locking
> considerations of this whole scheme. Linux, like most Unix systems,
> has never allowed hard links to directories for a number of reasons;
The claim is wrong. UNIX systems have traditionally allowed the
superuser to create hard links to directories. See link(2) for 2.10BSD
<http://www.freebsd.org/cgi/man.cgi?query=link&sektion=2&manpath=2.10+BSD>.
Having got that wrong throws doubt on the argument; perhaps a path can
simultaneously be a file and a directory.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [14]
More majordomo info at http://vger.kernel.org/majordomo-info.html [15]
From: Al Viro <viro@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[15]Date: Oct 18, 4:47 pm 2007
On Fri, Oct 19, 2007 at 06:07:45AM +0930, David Newall wrote:
> >considerations of this whole scheme. Linux, like most Unix systems,
> >has never allowed hard links to directories for a number of reasons;
>
> The claim is wrong. UNIX systems have traditionally allowed the
> superuser to create hard links to directories. See link(2) for 2.10BSD
> <http://www.freebsd.org/cgi/man.cgi?query=link&sektion=2&manpath=2.10+BSD>.
> Having got that wrong throws doubt on the argument; perhaps a path can
> simultaneously be a file and a directory.
Learn to read. Linux has never allowed that. Most of the Unix systems
do not allow that. Original _did_ allow that, but at the cost of very
easily triggered fs corruption (and it didn't have things like rename(2) -
it _did_ have userland implementation, of course, in suid-root mv(1),
but that sucker had been extremely racy and could be easily used to
screw filesystem to hell and back; adding rename(2) to the set of primitives
combined with multiple links to directories leads to very nasty issues on
_any_ system).
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [16]
More majordomo info at http://vger.kernel.org/majordomo-info.html [17]
From: David Newall <david@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[17]Date: Oct 18, 10:57 pm 2007
Al Viro wrote:
> On Fri, Oct 19, 2007 at 06:07:45AM +0930, David Newall wrote:
>
>>> considerations of this whole scheme. Linux, like most Unix systems,
>>> has never allowed hard links to directories for a number of reasons;
>>>
>> The claim is wrong. UNIX systems have traditionally allowed the
>> superuser to create hard links to directories. See link(2) for 2.10BSD
>> <http://www.freebsd.org/cgi/man.cgi?query=link&sektion=2&manpath=2.10+BSD>.
>> Having got that wrong throws doubt on the argument; perhaps a path can
>> simultaneously be a file and a directory.
>>
>
> Learn to read. Linux has never allowed that. Most of the Unix systems
> do not allow that.
I did read the claim and it is ambiguous, in that it can reasonably be
read to mean that most UNIX systems never allowed such links, which is
wrong. All UNIX systems allowed it until relatively recently.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [18]
More majordomo info at http://vger.kernel.org/majordomo-info.html [19]
From: Al Viro <viro@...>
Subject: Re: [RFC PATCH 0/5] Shadow directories
[19]Date: Oct 19, 1:37 am 2007
On Fri, Oct 19, 2007 at 12:27:16PM +0930, David Newall wrote:
> >Learn to read. Linux has never allowed that. Most of the Unix systems
> >do not allow that.
>
> I did read the claim and it is ambiguous, in that it can reasonably be
> read to mean that most UNIX systems never allowed such links, which is
> wrong. All UNIX systems allowed it until relatively recently.
FVO"relatively recently" exceeding a decade and half. In any case,
it's _trivial_ to get fs corruption on any system with such links -
play with rename() races a bit and you'll get it. And yes, it does
include 4.4BSD and quite a chunk of even later history.
Anyway, you are quite welcome to propose a sane locking scheme capable
of dealing with that mess.
As for the posted patch, AFAICS it's FUBAR in handling of .. in such
directories. Moreover, how are you going to keep that shadow tree
in sync with the main one if somebody starts doing renames in the
latter? Or mount --move, or...
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [20]
More majordomo info at http://vger.kernel.org/majordomo-info.html [21]Related links:
- Archive of above thread [21]
- Archive of above thread [21]