In the debate following Andrew Morton [interview] posting his plans for 2.6.13 [story], the existence of a plugin layer in Reiser4 was discussed. Jeff Garzik put it blunty, "the plugin stuff is crap. This is not a filesystem but a filesystem new layer. IMO considered in that light, it duplicates functionality elsewhere." Andrew Morton went on to explain, "I think the concern here is that this is implemented at the wrong level. In Linux, a filesystem is some dumb thing which implements address_space_operations, filesystem_operations, etc."

Hans Reiser noted, "please remember that this is per file, per item, per node, per attribute, per disk format, per bitmap, per super block, etc., abstracting, not per filesystem abstracting." He explained a couple advantages to plugins being that it makes it much easier for developers to change the disk format, and allows for easy code reuse. He added, "the use of plugins forced all the programmers to think about reusability at every layer of design. V3 of reiserfs is way too hard to work on and modify. If you ask one of the team to code something for V3 instead of V4, they quietly groan at the thought. It is just so much easier to do in V4."

Andrew Morton replied, "advanced features such as those which you describe are implemented on top of the filesystem, not within it. reiser4 turns it all upside down. Now, some of the features which you envision are not amenable to above-the-fs implementations. But some will be, and that's where we should implement those." The lengthy discussion continued, an interesting read for Reiser4 supporters and detractors alike.

From: Hans Reiser [email blocked]
To: Jeff Garzik [email blocked]
Subject: Re: -mm -> 2.6.13 merge status
Date:	Tue, 21 Jun 2005 13:05:44 -0700

Jeff Garzik wrote:

>
>
>> reiser4
>>
>>    
>
>
> The plugin stuff is crap.  This is not a filesystem but a filesystem +
> new layer.  IMO considered in that light, it duplicates functionality
> elsewhere.


What functionality where?  Please remember that this is per file, per
item, per node, per attribute, per disk format, per bitmap, per super
block, etc.,  abstracting, not per filesystem abstracting.

Plugins allow a number of things:

1) They allow us to never pay the cost to change the disk format again,
no matter how much we add in future years.  This really matters: the
prohibitive cost of disk format changes are the number one impediment to
filesystem improvements, and the primary reason why most filesystems
ossify after time has past.

2) They allow us to easily structure code for reuse.   If we want to
create a new kind of file that is like some other kind of file except
for one thing, we just write the one thing, and then easily reuse all
the other code, and create a new plugin id. 

The use of plugins forced all the programmers to think about reusability
at every layer of design.  V3 of reiserfs is way too hard to work on and
modify.  If you ask one of the team to code something for V3 instead of
V4, they quietly groan at the thought.  It is just so much easier to do
in V4.

When I asked one of our team to completely change the key assignment
algorithm for V4 (which controls what things get packed near what in the
tree), he complained that it would take 6 weeks to do it.  Under V3 it
would have taken 3 months.  It took him 3 days, and now to change it
again would take him 3 hours I think.  Oh, by the way, the change
boosted performance dramatically.

If you take the time to become familiar with coding within our plugin
layer, I think you will find yourself wanting the same to exist for
other filesystems.  Of course, no other filesystem needs to be impacted
by our plugin layer, and there is no way reiser4 could easily be
rewritten to exist without it (it would be like requiring that the
kernel not use function calls and only use gotos). 

Reiser4's plugin layer has as its explicit objective making it possible
for the weekend hacker to add something useful to reiser4 and send it in
for inclusion.  We want to democratize filesystem innovation so that
random bright guys who usually work on something other than filesystems
can act on their bright ideas without spending 3 years in the filesystem
field to do it.  I believe that most great filesystem innovations are
envisioned by persons not working on filesystems, and go nowhere because
of the especially high cost of entry into our field.

I am not as bright as other filesystem developers, and so we must tinker
with three ideas and keep one of them.  The speed of tinkering is
crucial to us, and the plugin layer increases that speed several fold.

Please reconsider your view. 


From: Christoph Hellwig [email blocked]
Subject: Re: -mm -> 2.6.13 merge status
Date:	Tue, 21 Jun 2005 21:24:48 +0100

Hans, we had this discussion before.  And the consensus was pretty simple:
the disk structure plugins are your business and fine to keep.  The
higher-level pluging that just add another useless abstraction below
file_operation/inode_operation/etc.  are not.  keep the former and kill
the latter and you're one step further.



From: Hans Reiser [email blocked]
Subject: reiser4 plugins
Date:	Tue, 21 Jun 2005 18:07:58 -0700

Christoph,

Reiser4 users love the plugin concept, and all audiences which have
listened to a presentation on plugins have been quite positive about
it.  Many users think it is the best thing about reiser4.  Can you
articulate why you are opposed to plugins in more detail?  Perhaps you
are simply not as familiar with it as the audiences I have presented
to.  Perhaps persons on our mailing list can comment.....

In particular, what is wrong with having a plugin id associated with
every file, storing the pluginid on disk in permanent storage in the
stat data, and having that plugin id define the set of methods that
implement the vfs operations associated with a particular file, rather
than defining VFS methods only at filesystem granularity?

What is wrong with having an encryption plugin implemented in this
manner?  What is wrong with being able to have some files implemented
using a compression plugin, and others in the same filesystem not.

What is wrong with having one file in the FS use a write only plugin, in
which the encrypion key is changed with every append in a forward but
not backward computable manner, and in order to read a file you must
either have a key that is stored on another computer or be reading what
was written after the moment of cracking root?

What is wrong with having a set of critical data files use a CRC
checking file plugin?

What we have hurts no one but us.  I have never seen an audience for one
of my talks that thought it hurt us..... most audiences think it is
great.  

Let us tinker with our FS, and you tinker with yours, and so long as
what we do does not affect your FS, let the users choose.

In the end, somebody will write a new fs that steals the good ideas from
both of us, and obsoletes us both.  They can only do this though if we
are left to be both free to implement differing filesystem designs.

I do not tell you how to design XFS, why are you making my life unpleasant?

Christoph Hellwig wrote:

>Hans, we had this discussion before.  And the consensus was pretty simple:
>the disk structure plugins are your business and fine to keep.  The
>higher-level pluging that just add another useless abstraction below
>file_operation/inode_operation/etc.  are not.  keep the former and kill
>the latter and you're one step further.
>
>
>  
>



From: Jeff Garzik [email blocked]
Subject: Re: reiser4 plugins
Date:	Tue, 21 Jun 2005 21:14:06 -0400

Hans Reiser wrote:
> Christoph,
> 
> Reiser4 users love the plugin concept, and all audiences which have
> listened to a presentation on plugins have been quite positive about
> it.  Many users think it is the best thing about reiser4.  Can you
> articulate why you are opposed to plugins in more detail?  Perhaps you
> are simply not as familiar with it as the audiences I have presented
> to.  Perhaps persons on our mailing list can comment.....
> 
> In particular, what is wrong with having a plugin id associated with
> every file, storing the pluginid on disk in permanent storage in the
> stat data, and having that plugin id define the set of methods that
> implement the vfs operations associated with a particular file, rather
> than defining VFS methods only at filesystem granularity?

You're basically implementing another VFS layer inside of reiser4, which 
is a big layering violation.

This sort of feature should -not- be done at the low-level filesystem level.

What happens if people decide plugins are a good idea, and they want 
them in ext3?  We need massive surgery to extract the guts from reiser4.

	Jeff



From: David Masover [email blocked]
Subject: Re: reiser4 plugins
Date:	Tue, 21 Jun 2005 23:25:24 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Garzik wrote:
> Hans Reiser wrote:
> 
>> Christoph,
>>
>> Reiser4 users love the plugin concept, and all audiences which have
>> listened to a presentation on plugins have been quite positive about
>> it.  Many users think it is the best thing about reiser4.  Can you
>> articulate why you are opposed to plugins in more detail?  Perhaps you
>> are simply not as familiar with it as the audiences I have presented
>> to.  Perhaps persons on our mailing list can comment.....
>>
>> In particular, what is wrong with having a plugin id associated with
>> every file, storing the pluginid on disk in permanent storage in the
>> stat data, and having that plugin id define the set of methods that
>> implement the vfs operations associated with a particular file, rather
>> than defining VFS methods only at filesystem granularity?
> 
> 
> You're basically implementing another VFS layer inside of reiser4, which
> is a big layering violation.

There's been sloppy code in the kernel before.  I remember one bit in
particular which was commented "Fuck me gently with a chainsaw."  If I
remember correctly, this had all of the PCI ids and the names and
manufacturers of the corresponding devices -- in a data structure -- in
C source code.  It was something like one massive array definition, or
maybe it was a structure.  I don't remember exactly, but...

The point is, this was in the kernel for quite awhile, and it was so
ugly that someone would rather be fucked with a chainsaw.  If something
that bad can make it in the kernel and stay for awhile because it
worked, and no one wanted to replace it -- nowdays, that database is
kept in userland as some nice text format -- then I vote for putting
Reiser4 in the kernel now, and ironing the sloppiness ("violation") out
later.  It runs now.

> This sort of feature should -not- be done at the low-level filesystem
> level.

I agree there, too.  In fact, some people have suggested that all
"legacy" (read: non-reiser) filesystems should be implemented as Reiser4
plugins, effectively killing VFS.*

So, Reiser4 may eventually take over VFS and be the only Linux
filesystem, but if so, it will have to do it much more slowly.  Take the
good ideas -- things like plugins -- and make them at least look like
incremental updates to the current VFS, and make them available to all
filesystems.

Eventually, this would result in a full merge of Reiser and Linux, such
that the only thing left of "Reiser4" are a few plugins -- things like
storage plugins and maybe some more exotic things like fibration that I
don't really understand.

> What happens if people decide plugins are a good idea, and they want
> them in ext3?  We need massive surgery to extract the guts from reiser4.

And here is the crucial point.  Reiser4 is usable and useful NOW, not
after it has undergone massive surgery, and Namesys is bankrupt, and
users have given up and moved on to XFS.  But the massive surgery should
happen eventually, partly to make all filesystems better (see below),
and partly to make the transition easier and more palatable for those
who don't work for Namesys.





* Imagine ext3 as a storage-level plugin for reiser4.  You get one
benefit immediately -- lazy allocation.  Lazy allocation is nice both
for fragmentation and for busy systems writing and nuking a lot of
temporary files.  Imagine a file which is written and then deleted
before it hits disk -- it should never touch the disk, regardless of the
underlying storage layout.

Other benefits are subtler but inevitable.  Right now, it would be an
understatement to say that there's duplication of effort between ext3,
xfs, and reiser4.  And yet, I don't think there are many core design
decisions that influence my decision as to which I pick up.  I just want
it to be fast, stable, and have a stable on-disk format so I don't have
to reformat too often.  I honestly don't care about how XFS is
segmenting the disk, or Reiser4 packs really well, or ext3 can be read
as ext2 and flushes to disk every five seconds.  These are the kinds of
things which should be set to sane defaults, tunable for enterprise
users, but are not really enough to warrant completely separate code bases.

I would imagine that it wouldn't be too long after this before an
uber-fs rose, something which combined enough of the strong points of
all the existing Linux filesystems that no one would use anything else.
 But, Linux still needs support for FAT32, ISO9660, UDF, and all the
other filesystems which won't go away as easily as XFS and ext3, and it
would be nice if these could all share as much code as possible.


I don't know if storage plugins really work that way, but they should.

I think.  I don't work here.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrjoNHgHNmZLgCUhAQIYYw/7BWZ0gVvy0ln5tRo405yUoRHJ/jVFBGyP
5pQ610ECMZORVWRO1qP/NXbvGZwDjEggM5iIeahsGqnBWzNGDsB6RslMUZAxzCYy
iAovi0881zoARf3dmhKrDgbGkvNLPTx+/ypa20oRcHLnyI+NAjerUxNuvGc79PJn
Ybm9JhX6tQsqGIKjZye9uNDHCifLqzA1gdxucPwWo9sz4ymzM9FgsMGvb+IxrU50
2a2G2K6AHcH+nkomEHw3xgY3PmUZUy0s2hQDSkJx6dCido7fGZwwykaSXm4IZs9s
xZqBxKx32G/LDnDV3W8gpj8ZisUE+58kefRbIjo4Ml6IzgXvQqpRjaQOuz8JoKDX
9KUV43tcZkPpK+neIYPQYpXCrdMQqB5+ISpbNKuz/Z/abkcVw1042sy0EKM+/VnD
n3Jr7PpSyk0lfCyADr+33qnqPQRFq02gQTohg9FheoMthhV01aaeGW5ExmTM/Wwa
6Dmv/qnn2oNImi+Ebz5u3Lbnz7lL3MVpRL4aoMmEOVUB3xhehnxesf//yacBmwj9
M/3KVae9epwX4K6yi8qQXzH4160IBFHpWUxBLc9LnOZlHQuZ+Fz3BIrbKQBvmaHT
7lrwi0Os6TmiBGMSFd6OUWHcYs4p97aMw30NG33fCL6e8X6oNVFwwnJXZpBPN1Mr
+lrsVAywKEI=
=rHdK
-----END PGP SIGNATURE-----



From: Jeff Garzik [email blocked]
Subject: Re: reiser4 plugins
Date:	Wed, 22 Jun 2005 01:18:52 -0400

David Masover wrote:
> There's been sloppy code in the kernel before.  I remember one bit in
> particular which was commented "Fuck me gently with a chainsaw."  If I
> remember correctly, this had all of the PCI ids and the names and
> manufacturers of the corresponding devices -- in a data structure -- in
> C source code.  It was something like one massive array definition, or
> maybe it was a structure.  I don't remember exactly, but...
> 
> The point is, this was in the kernel for quite awhile, and it was so
> ugly that someone would rather be fucked with a chainsaw.  If something
> that bad can make it in the kernel and stay for awhile because it
> worked, and no one wanted to replace it -- nowdays, that database is
> kept in userland as some nice text format -- then I vote for putting
> Reiser4 in the kernel now, and ironing the sloppiness ("violation") out
> later.  It runs now.

Existence of ugly code is not an excuse to add more.

We have to maintain said ugly code for decades.  Maintainability is a 
big deal when you deal with the timeframes we deal with.


> So, Reiser4 may eventually take over VFS and be the only Linux
> filesystem, but if so, it will have to do it much more slowly.  Take the
> good ideas -- things like plugins -- and make them at least look like
> incremental updates to the current VFS, and make them available to all
> filesystems.

This is how all Linux development is done.

The code evolves over time.

You have just described the path ReiserFS needs to take, in order to get 
this stuff into the kernel, in fact.


> And here is the crucial point.  Reiser4 is usable and useful NOW, not
> after it has undergone massive surgery, and Namesys is bankrupt, and
> users have given up and moved on to XFS.  But the massive surgery should
> happen eventually, partly to make all filesystems better (see below),
> and partly to make the transition easier and more palatable for those
> who don't work for Namesys.

We care about technical merit, not some random company's financial 
solvancy.  Reiser4 has layering violations, and doesn't work with the 
current security layer.  Those are two biggies.

There is no entitlement to be merged in the upstream kernel.  If people 
don't like how the Linux kernel is managed, they are free to maintain 
their own fork.  If its better, people will use that instead.


> I would imagine that it wouldn't be too long after this before an
> uber-fs rose, something which combined enough of the strong points of
> all the existing Linux filesystems that no one would use anything else.
>  But, Linux still needs support for FAT32, ISO9660, UDF, and all the
> other filesystems which won't go away as easily as XFS and ext3, and it
> would be nice if these could all share as much code as possible.
> 
> 
> I don't know if storage plugins really work that way, but they should.

No, they shouldn't.


> I think.  I don't work here.

Obviously.

	Jeff



From: Andrew Morton [email blocked]
Subject: Re: reiser4 plugins
Date:	Tue, 21 Jun 2005 18:18:02 -0700

Hans Reiser [email blocked] wrote:
>
> What is wrong with having an encryption plugin implemented in this
>  manner?  What is wrong with being able to have some files implemented
>  using a compression plugin, and others in the same filesystem not.
> 
>  What is wrong with having one file in the FS use a write only plugin, in
>  which the encrypion key is changed with every append in a forward but
>  not backward computable manner, and in order to read a file you must
>  either have a key that is stored on another computer or be reading what
>  was written after the moment of cracking root?
> 
>  What is wrong with having a set of critical data files use a CRC
>  checking file plugin?

I think the concern here is that this is implemented at the wrong level.

In Linux, a filesystem is some dumb thing which implements
address_space_operations, filesystem_operations, etc.

Advanced features such as those which you describe are implemented on top
of the filesystem, not within it.  reiser4 turns it all upside down.

Now, some of the features which you envision are not amenable to
above-the-fs implementations.  But some will be, and that's where we should
implement those.



From: Andi Kleen [email blocked]
Subject: Re: reiser4 plugins
Date:	22 Jun 2005 03:26:26 +0200


First Hans let me remind you that this discussion is not XFS vs
reiser4. Christoph does a lot of reviewing and your child definitely
is in serious need of that to be mergeable. I'm sure Christoph is able
to review inpartially even when he is involved with other FS.

Jeff Garzik [email blocked] writes:
> 
> You're basically implementing another VFS layer inside of reiser4,
> which is a big layering violation.
> 
> This sort of feature should -not- be done at the low-level filesystem level.
> 
> What happens if people decide plugins are a good idea, and they want
> them in ext3?  We need massive surgery to extract the guts from
> reiser4.

We already kind of have them, there are toolkits to do stackable FS with
the existing VFS.

However I suspect Hans is unwilling to redo his file system in this
point. While it looks quite unnecessary there might be other
areas which deserve more attention. In general all the code
needs review, even if it is not as controversal as the reinvented VFS.

-Andi



From: Hans Reiser [email blocked]
Subject: Re: reiser4 plugins
Date:	Tue, 21 Jun 2005 19:47:13 -0700

Andi Kleen wrote:

> Christoph does a lot of reviewing 
>
and he is notorious for making needed linux contributors go away and not
come back, and I won't say which famous person on this mailing list told
me that....

>and your child definitely
>is in serious need of that to be mergeable. I'm sure Christoph is able
>to review inpartially even when he is involved with other FS.
>  
>
As impartial as a puppy on PCP....

Christoph is aggressive about things he does not take the time to
understand or ask about first.  I hate that.   I wish he would go away
please.  He is not exactly an Ousterhout, Rob Pike, Granger, Mazieres,
Frans Kaashoek, etc.,  in his accomplishments, so why is he reviewing
other people's filesystems?  Reviews are great, how about finding
persons who have created filesystem innovations (and thus are less
likely to reject innovations without understanding them) to do them? 

How about review by benchmark instead?

It works, it runs faster than the competition, users like it, we
addressed the core kernel patch complaints, it should go in and receive
the exposure that will result in lots of useful improvements and
suggestions.   It seems like we are getting an unusual review process. 

I used a bunch of algorithms for which there was a consensus among those
consulted that the algorithms were a bad idea, only the fact that I paid
for the code to be written and required that it be done my way (ignoring
the "you just use me as a typewriter" remarks as best I could)  caused
the algorithms to get implemented at all, and the benchmarks then proved
the algorithms were a good idea.  V3 performance sucks in major part
because I listened to the consensus when I knew better.   I don't really
care for consensus on my FS anymore.   If you guys want to understand
what I am doing I am happy to talk about it extensively, but please
don't require that I groupthink.  I frankly think that with my
benchmarks, I should be allowed to tinker on my own. 

Hans The Mad



From: Kyle Moffett [email blocked]
Subject: Re: reiser4 plugins
Date:	Tue, 21 Jun 2005 23:16:31 -0400

On Jun 21, 2005, at 22:47:13, Hans Reiser wrote:
> Andi Kleen wrote:
>> and your child definitely
>> is in serious need of that to be mergeable. I'm sure Christoph is  
>> able
>> to review inpartially even when he is involved with other FS.
> As impartial as a puppy on PCP....

So where else are you planning to get a competent reviewer who is fluent
in the internals of filesystems?  Good reviewers don't grow on trees,  
and
in order to be able to understand filesystem issues, one must generally
have worked with them before...  Besides, what good is insulting others
going to do?

> Christoph is aggressive about things he does not take the time to
> understand or ask about first.
[rant snipped]

 From my objective re-reading of his posts, I can see that he is  
critical
of things that are difficult to understand not just to be critical, but
to provoke additional thought over those portions of the code.  In many
cases this leads to better abstractions and simpler code than otherwise.

> How about review by benchmark instead?

/dev/sda is a great filesystem with awesome benchmarks, assuming one  
only
needs to store a single file.  Besides, benchmarks aren't the only thing
important about code.  If the interface consists of:

   void clear_current_filename(void);
   void add_char_to_current_filename(char x);
   void read_bytes_from_current_file(char *byte, unsigned long size);
   void write_bytes_to_current_file(const char *byte, unsigned long  
size);

then this is clearly not a good API, regardless of how well or poorly it
may perform.

> It works, it runs faster than the competition, users like it, we
> addressed the core kernel patch complaints, it should go in and  
> receive
> the exposure that will result in lots of useful improvements and
> suggestions.   It seems like we are getting an unusual review process.

If you look over other patches in -mm, you will see that your review
process is not unusual, especially given the number of concerns that  
other
developers have raised over Reiser4.

[irrelevant algorithm rant snipped]

> If you guys want to understand
> what I am doing I am happy to talk about it extensively, but please
> don't require that I groupthink.  I frankly think that with my
> benchmarks, I should be allowed to tinker on my own.

Yes, you can tinker on your own all you want.  Another project that has
taken that route is GrSecurity, which was alive and well last I checked.

If you don't like others criticisms, take up your marbles and go home,
just don't expect them to accept your work when you've not fixed it to
community standards.

Cheers,
Kyle Moffett

--
Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't.  
That's why
I draw cartoons. It's my life."
-- Charles Shultz

• Archive of above thread
• KernelTrap interview with Andrew Morton