(Andrew, perhaps this could bake in -mm for a cycle or two if

there are no objections?)
These patches 1-7 add the Optimized MPEG Filesystem, a proprietary

filesystem used by the embedded devices Rio Karma and ReplayTV

which are no longer manufactured.
I've been maintaining this reverse-engineered filesystem

out-of-tree since around 2.6.12.  It has stabilized to the point

that, while not quite feature-complete, it is close enough that

I'm pretty much just applying VFS API updates to it these days.

So, in the spirit of stable_api_nonsense, I request that it be

considered for inclusion for .26 or later.  
There is a small user community (<20?) using the driver today.

According to google, it has been packaged by others for Debian,

Ubuntu, and Mandriva.
I have also written an implementation in FUSE.  However, in

its current unoptimized state, the FUSE version is about half the

speed of the kernel driver.  Either way, the kernel version could

use a review given some distros have apparently picked it up.
version 3:

 - fixed sparse warnings

 - added more sanity checking in list traversals
version 2:

 - removed private inode cache

 - included GPL v2 notice

 - rewrote omfs_count_free to use hamming weight library functions

 - removed duplicated crc-itu-t
Version 1: initial post
--

Adding a new FS to Linux is a pretty major thing.
- A key question (which you don't seem to have addressed at all!) is: why

  is this a useful addition to Linux?  What are the filesystem's strengths?

  What is its application?  How does it improves Linux and by how much?
  I'll go aehad and assume that its sweet-spot is streaming media files

  (perhaps more than one at a time?).  If so, then comparative performance

  measurements would be the key piece of information which we'll need when

  making a merge decision.  Probably against ext3, ext4 and XFS.
- What are the filesystem's features?  Does it journal changes, or is it

  inherently crash-safe?  Or is fsck-style repair needed?
- We'd like to see some documentation:
  - Mount options
  - Location of userspace support tools such as mkfs and fsck
  - Documentation for those tools
- A MAINTAINERS record, please?
If I include this in -mm, some kind people will try it out.  It would be

most helpful for the changelog (or even the Kconfig help) to include

sufficient information for people to be able to create, mount and check an
Who did the reverse-engineering, and how was it done?
Please make us confident that we won't get our butts sued off or something.

--

Honestly, I'm not sure if this FS is mainline material or not.  If it is,

it is in the sense that the Amiga FS, befs, and so forth are useful:

it's a dead filesystem that a very few people still have a reason to

use.  If FUSE is where this should live, then I'll just simply focus my

time on that instead (since I already have it in FUSE).
Primarily the users are Rio Karma owners such as myself.  For some reason

the designers stuck this filesystem on the device, so unless someone hacks

the firmware, using OMFS is the only way to get MP3 files on the thing 
Okay.  I can do benchmarks but I'm pretty sure it will be worse than all

of them by a long shot.  
Note my Kconfig text:

[...]

+	  player and ReplayTV DVR.  Despite the name, this filesystem is not

+	  more efficient than a standard FS for MPEG files, in fact likely

+	  the opposite is true.  
Uncached lookups in particular are very slow because directories are

essentially a huge hash table.  You might have to seek through several

siblings in a bucket to get to the target file.
They seem to have designed for online recovery with checksumming and

mirroring of every FS object.  This module updates these, but it does
I'm not sure there's a company to sue anymore.
Most of it was done by members of the ReplayTV community; I just used

their documentation.  I'll see if I can get in touch with one of them to

see how they came up with that.  I only figured out a few things, for

which I'd use the Karma as a black box for copying/deleting files etc.

and then poke at the resulting disk image with a hex editor.
--

Bob Copeland %% www.bobcopeland.com 
--

<IANAL>
Bob, you have to be careful there; an argument like "there may not be a

company to sue" may not hold.  IP rights, patents and their filings, and

more, can be sold, traded, transferred, inherited, etc. -- even if the

original company is no longer in business.  I don't know the history of

ReplayTV or how it went out of business, but it might be worth a check to

see if there's anyone who can still claim any rights over any part of OMFS

(e.g., chapter 11 creditors who may feel they didn't get fully compensated).

The irony is that as long as omfs gets little attention, no one is likely to

sue; but once and if it gets into mainline and lots of Big Name Linux

distros start carrying it, *then* someone may come out from the shadows to

sue.
</IANAL>
Erez.

--

This is actually quite interesting. Checksuming filesystem sounds

pretty useful.
--

(english) http://www.livejournal.com/~pavelmachek

(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--

> Honestly, I'm not sure if this FS is mainline material or not.  If it is, 
It looks mainline to me. It is far more relevant than most of the other

file systems we support and it doesn't mess up core code at all. It's a

loadable kernel module so the impact is basically nil, as well as being
If you used their documentation and didn't do the reverse engineering or

read any ReplayTV code there shouldn't be any problems. The big danger is

if you reverse engineer their code and you write new code you may copy

their implementation even accidentally (or be held to have done so).
Alan

--

Yes, pursuing the FUSE implementation sounds a better approach - it avoids

burdening the kernel with a filesysstem which few will be interested in and

is more practical for use by those who _are_ interested in it.
--

No way.  For a normal foreign block filesystem a proper kernel

implementation is much better.  And this one is particularly

well-written.  Lately I really start wondering why we keep adding crap

all over the core, but if we have a modular new filesystem that's quite

---end quoted text---

--

I'm not complaining about anything.  Who has?
As the filesystem is for occasional, non-performance-sensitive use

by a very small number of people, doing it via FUSE sounds like an

all-round more practical approach.  This has nothing to do with quality of

implementation at all.
I don't have particularly strong opinions either way.
--

It's a stupid idea.  Moving a simple block based filesystem means it's

more complicated, less efficient because of the additional context

switches and harder to use because you need additional userspace

packages and need to setup fuse.
We made writing block based filesystems trivial in the kernel to grow

more support for filesystems like this one.

--

FUSE is more than just providing a framework for block based filesystems.

Sometimes FUSE is criticised for the extra useful features it provides. 
This is about as true as claiming FUSE doesn't have any context switch

overhead. Sometimes it has none, sometimes small, sometimes it can be

significant (e.g. when not using the 50 fold context switch speed up

patch). The question is, how releavant it is? Just some short notes,
 1. Using the in-kernel cached data involves no context switch for FUSE.
 2. On commodity hardware Linux can do a million context switches.

    File system workloads barely need or can do more than a few tens

    of thousands file operations per second (fsops) due to storage

    bottlenecks. Which means maximum about only extra 5% CPU use for

    block based FUSE file systems. 
    If they do more fsops then typically it's served from the kernel caches

    and no user space and context switches are involved at all.
 3. ext3 with the highly optimized dir_index is far the fastest

    traditional block based file system in file creation. Once I wrote

    a FUSE file system which apparently would have been faster if the

    VFS wouldn't do needlessly a lookup() before create(). AFAIK some

    network file systems have the some performance problem because of

    this.
 4. The dominant factors for performance is design, quality of the

    implementation and lead time to optimize for the (latest) hardware.

    What's the best way to realize this depends on many factors.
 5. Some workloads can indeed trigger very high context switches. This

    could be improved/solved but probably it would be more beneficial to 
The fuse install should solve all setup issues and a fuse fs can be written

where the traditional commands work:
	mount -t fstype device mountpoint

	umount mountpoint
Ntfs-3g doesn't even need fuse user space being installed, only a

modprobeable fuse kernel module.
	Szaka
--

NTFS-3G:  http://ntfs-3g.org

--

* Szabolcs Szakacsits <szaka@ntfs-3g.org> wrote:
Hi folks,
<big_snip />
I've just fought through this thread, maybe missed some points,

and I'll try no to repeat to repeat arguments already said, but:
IMHO, many of the filesystems should belong to userland,

(especially those I'd call "exotic"). Maybe some decision points:
* high complexity (-> not easy to debug)

* not required for booting

* not yet matured (within kernel)

* not performance critical

* requires much userland assistance

* not used permanently (just from time to time)

* not actually an IPC mechanism
My first candidate would be coda: 
It never worked well for me, even often have to reload the

kernel driver. As most of the logic already *is* in userland,

there wouldn't be a performance tradeoff when doing the kernel

interface entirely via FUSE or even let venus just be an 9P server.
cu

--

---------------------------------------------------------------------

 Enrico Weigelt    ==   metux IT service - http://www.metux.de/

---------------------------------------------------------------------

 Please visit the OpenSource QM Taskforce:

 	http://wiki.metux.de/public/OpenSource_QM_Taskforce

 Patches / Fixes for a lot dozens of packages in dozens of versions:

	http://patches.metux.de/

---------------------------------------------------------------------

--

I don't feel strongly either way, and Christoph's arguments against

fuse are mostly valid (although neither of them are serious).
There's one thing which makes fuse a slightly better candidate for

applications where the number of users is low: stability.  Unless you

or your users test the hell out of your filesystem, there always a

chance that some bugs will remain.  These rarely bring down the whole

system, but it usually requires a reboot to let you continue using the

Oopsing fs.
Miklos

--

I don't have hard numbers, but anecdotally my FUSE version is quite

a bit less performant.  That's no criticism of FUSE - I just haven't
Sure, though this FS won't see the same kind of use as ext2.  Most users

would just mount it, copy a bunch of files, then unmount it, and if that

works then great.
It has at least seen some testing with fsx, though I had to turn off most

of the checks since growing truncate is still unimplemented.  
--

Bob Copeland %% www.bobcopeland.com 
--

Thankfully you need none, it's already there by FUSE and the kernel. The

trick is exactly that you can have the kernel performance and the left is

moved to user space with typically negligible performance overhead which is

usually well compensated with faster delivered new features and bug fixes.
The completely unoptimized ntfs-3g read/write saturates my USB disks with

25-30 MB/sec using 8% (read), 35% (write) CPU time on a 2.5 GHz Core 2 Duo

with cold caches.
If you have the free hot caches then it performs the same as in-kernel

file systems, user space isn't involved at all.
I noticed that the OMFS kernel driver supports only the USB interface and

the FUSE one both the network and the USB one. Isn't it possible that you

compared the performance using the USB with the kernel vs the much slower

and lower latency network with FUSE? You should compare performances using

only the USB interface in both cases.
If you did use the USB interface with FUSE then what exactly do you mean by

"quite a bit less performance" in numbers and workloads? What you did, how

long it took using what CPU?
Thanks,

	    Szaka
--

NTFS-3G:  http://ntfs-3g.org

--

Correct me if I'm wrong, but one place where caches seem necessary is for

lookup.  My file system already has an inode number; my understanding

is that the kernel inode cache and dcache are caching the FUSE inode by

filename and its hashed inode number.
In FUSE, on open, I'm passed a filename which I then have to resolve into an

inode # via my own lookup.  The VFS does the path_lookup as part of sys_open,

and since I get to put private data into the struct inode, I'll generally

already have the block # and various other info in the dcache by the time

open is called.
Also, if you stuff inode data into the private fh field in fuse_file_info,

you need to be sure that any subsequent lookups always return the same

inode structure, otherwise a thread doing ftruncate vs one doing truncate

will cause issues.  So I created an internal dcache to solve those two
Nope, that's not possible, sorry.  Both require use of USB.  lkarmafs and
Like I said it was anecdotal (copy 20 gigs of X) in both.  I'm sure a

good portion of it is my fault, such as doing unnecessary malloc & copies

in omfs_fuse.  I have put exactly zero effort into making it fast so far.
BTW, I hardly intended to start a huge VFS vs FUSE debate.  I think FUSE

is great.  I'm not sure it's the right fit for this, is all.
--

Bob Copeland %% www.bobcopeland.com

--

I also don't advocate any solution, only interested in the FUSE myth

busting and making it easier to use and develop for, i.e. having the best

possible performance with zero effort.
I checked both solutions quickly, I think they are nicely written.
The kernel driver had 30 MB/s with 10% CPU usage, the FUSE version had

6 MB/s with high I/O waiting.
The major reason seems to be that the FUSE version reads heavily from the

block device during pure write operations, while the kernel driver never.

There can be several non-exclusive explanations.
One of them, as you wrote, the functionality is not exactly the same, the

FUSE one does more. If it's relevant or not, you should know. If anything

is involved with reading from multiply places regularly then it's relevant.
Moreover when you're writing to a block device from user space then the

size and position of the block should be page aligned, otherwise you end up

doing unwanted synchronous reads instead of the believed asynchronous

writes. Solving this issue for the most common cases resulted a sometimes

over 10 fold write speedup in ntfs-3g. But of course it would be nice if

the kernel just provided this for everybody.
Btw, mkomfs.c is missing a '#define _FILE_OFFSET_BITS 64' which is needed

to open LFS files.
        Szaka
--

NTFS-3G:  http://ntfs-3g.org

--

Yep, I suspect switching to the lower-level API plus switching to mmap()

and getting rid of various memcpy's in omfs.c would make a huge difference.
Cool, thanks for the tips!
--

Bob Copeland %% www.bobcopeland.com

--

Fuse has two different APIs.  For the "high level", path based one,

this is true.  The "low level" one is very similar to the one provided

by the VFS.
On the low level interface, when the ->lookup() method is called, it

will provide a "nodeid", which is just a per-inode unique cookie (not

the same as st_ino).  It can be a pointer to the private inode data

for example, and in the ->forget() method this private data can be
And I think the VFS is great.  Undoubtedly kernel programming has it's

own charm, and I definitely don't want to scare you away from that.

Merging into mainline is a great reward, which must be erned the hard

way.  The debate is just part of that ;)
I also want to dispel any myths surrounding fuse, because those help

nobody.
Miklos

--

Sure.  I'll go on record saying that omfs_fuse was written over a weekend and

can not be considered a good example of anything.  I already had a

proto-libomfs that I had made for omfsck and mkomfs, and wanted to see how

easy it would be to wire that stuff up into a FUSE fs.  Pretty easily, it

turned out.
--

Bob Copeland %% www.bobcopeland.com

--

Presumably if "user space isn't involved at all", it must require that

user space has granted caching rights to the kernel over a FUSE

cache coherency protocol?
Otherwise I don't see how the kernel could coherently cache file pages

for some kinds of FUSE filesystems.  (E.g. sshfs, for example: every

operation must surely invoke a user space request or involve granting

a caching right to the kernel, to keep accesses coherent with other

users of the same remote files).
Ergo, either its not coherent, or there is some coherency protocol,

which does require _some_ work in the user space implementation.
-- Jamie

--

Most filesystems (ntfs-3g included) cannot be modified from the

outside, so coherency isn't an issue.
Otherwise currently fuse provides rather crude settings for caching:
  - timeout for attributes (per-inode, default 1s)

  - timeout for names (per-dentry, default 1s)

  - page cache is bypassed completely (per-file, default off)

  - invalidation of page cache on open (per-open, default on)
Writes are synchronous even in the caching case, for various reasons.

That is one of the reasons why write performance can be worse than

read performance.  For some filesystems (ntfs-3g), this limitation

could be lifted, but that requires that the mtime handling be moved to

the kernel in these cases, which is not yet possible.
There are also plans to add some sort of cache coherency protocol,

where the filesystem can asynchronously call back to fuse to
Sshfs is not coherent (but neither is NFS), it just has timeouts for

caches and invalidation based on modification time.
Miklos

--

Seems sensible.  As someone who is deep in coherency protocols at the

moment (I'm writing a robust distributed database/filesystem) I don't
Great!
I suggest adding another option (as well) where the filesystem can ask

fuse to send it synchronous validation requests - some things require

that.  (In my own work, the choice of A->B async invalidation and B->A

synchronous validation is heuristic: some usage patterns benefit from
Fwiw, I think NFS version 4 is coherent (it uses leases), and older

NFS should be coherent when you use fcntl file locks (it's not very

efficient though).
I have been bitten a few times by timeout based caches in the past

(NFS and SMB (pre-oplock)).  Simple things like editing a file, then

running "ssh compiler-box make" from the editor quietly building

incorrect code - and even subsequent make commands don't fix it.  Or

when I edit a file, then tell someone I've changed the file - and then

they edit the file, and my edits are lost.  Very annoying.  Nobody

should build those kind of caches into new software.  :-)
-- Jamie

--

Yes, that makes sense.  I expect this could be done by extending the

existing requests with a flag saying the result is already cached.

And then the filesystem can either reply with a special "cached data
Oh well, you can turn off caching if it bothers you :)  OTOH it would

be rather hard (and probably against the point) to try to extend the

sftp protocol to handle cache coherency.  Sshfs is not meant to be a

normal filesystem (although some people are trying to use it for home

directories and such), just a simple way to access remote files.
Miklos

--

Oh, I agree.  Violent agreement, they call it :-)
-- Jamie

--

The worst I/O performance problems should be gone by 2.6.26.

Otherwise there shouldn't be a need to add optimizations to the

userspace code.  The kernel caches take care of that, just like for a
Exactly.  Which means, that bugs which happen only in special

circumstances don't surface early and cause more headaches later.
Miklos

--

I think it's a slippery slope from that to rewriting Linux as a

microkernel.
Simple block filesystems like this belong in the kernel.
--

dwmw2
--

You say that as if a microkernel had _no_ advantages.  Which isn't

true: it's just a trade between performance and encapsulation.  What I

was saying, that if there are few users, and so the tester base is

limited, then they _might_ just be better off with a slower, but more

stable solution.
I'm not advocating moving ext3 to fuse.  And I didn't advocate moving

ntfs to fuse, still that was done and the resulting filesystem at the

moment happens to outperform the kernel one in every respect ;)
Miklos

--

Yes, that's quite annoying as FUSE doesn't work with my architecture,

so I can't use the only good NTFS driver.... grr! :-)
FUSE is great for user-mounted filesystems, though.  Having to use

"su" in shell scripts which modify loopback filesystems was always

very ugly.
I don't see why we need such a dichotomy, though.  The VFS API is

sensible enough that it should be possible to compile the same

filesystem code to run in kernel or in FUSE.
-- Jamie

--

Well, yeah:
  http://lkml.org/lkml/2006/2/27/148
Oh, you mean without having to run the whole kernel around the

filesystem?  Well, that's a bit more tricky: filesystems use not just

the VFS API, but the block, VM, SLAB, etc. APIs as well.  Porting all

of these to userspace and keeping them in sync with the kernel changes

doesn't sound much fun.
Miklos

--

It's not really a joke.  It really is a problem: I'm supposed to

provide NTFS support for a project, but the kernel I'm using doesn't

work with FUSE (too old for the FUSE version needed by ntfs-3g), and

the in-kernel NTFS isn't good enough - all the active development work

is done on ntfs-3g.
The :-) is because it's my problem, nobody else's.
-- Jamie
--

How old?  The out of tree fuse module (hoping to get rid of it soon)

works down to 2.6.9.  Beyond that it does probably require a fair

amount of porting work.
Miklos

--

I imagine that backporting FUSE to 2.4 no-mmu will be more work than

backporting a filesystem, but I could be mistaken.  Moving my project

to a 2.6 kernel (*any* 2.6 kernel) would be much more work than

either.
All other filesystems I care about are in-kernel - NTFS is quite

exceptional in needing FUSE just to get ordinary, stable file access.

(The in-kernel NTFS is regarded as unstable and missing essential

basic features.)
None of this should be taken as any kind of criticism.  I love what

you guys have done.
-- Jamie
--

Fuse-2.5 still supported linux-2.4.  Although I have no idea if it

works with no-mmu, maybe it's worth a try.
Backporting ntfs-3g to an older version of fuse shouldn't be a big

problem, as the fuse interface didn't change very much since then.
Miklos

--

Thanks for your advice.  I will look into all these possibilities.

Fuse working would be quite useful in many other ways too.
-- Jamie

--

ntfs-3g was reported to work on 2.4 no-mmu kernels six months ago (though

it had no our QA). Please see http://ntfs-3g.org/support.html#kernel24 and

search for nommu on fuse-devel for the details.
	Szaka

--

Ah, wonderful!  2.4 kernel support has been _added_ to NTFS-3g's

"FUSE-lite" since I last looked!
Nice to see 2.4 kernel support is still an active priority ;-)
Thanks for the info, it's really useful.  I would never have thought

to check for 2.4 support being added back, since an earlier

announcement said it was no longer supported.
(Btw, The NTFS-3g release notes, on it's decision to replace the FUSE

userspace libs (which is what makes it work with 2.4 kernels too),

suggest there may be serious problems in the standard FUSE libs,

especially on small devices.  It doesn't bother me, as I only need

NTFS-3g, not other FUSE capabilities, but perhaps FUSE developers

would like to take a look at those problems.)
Thanks again,

-- Jamie

--

There were compatibility problems with uClibc, but those have now been

addressed in the standard fuse libs as well.
I actually think fuse-lite is a good thing (as long as someone is

maintining it): it allows dynamically changing projects like ntfs-3g

to distribute and link their own version of libfuse, which contains

the latest features they need.  Not having to wait for these features

to slowly make it into distributions can be a big advantage.
At the same time fuse-lite doesn't need to carry all the source and

binary compatibility cruft that the standard fuse lib needs, so

statically linking it has a relatively small overhead.
Miklos

--

It's getting a little offtopic, but what keeps you from backporting

the required fuse version?
--

That would be UML :)
But seriously -- wouldn't our time be better spent on improving error

handling in the kernel, rather than taking advantage of the benefits of

microkernel design... but only for those parts which don't matter?
--

dwmw2
--

Sure, but that needs reviewer time, tester time and developer time.
But if people do it for fun, or for having a simple example code to

look at, I have no problem with that :)
Miklos

--

Gad.  Why?

--

Hi,
Miklos has the wrong end of the stick.  No-one has "moved" ntfs to

fuse.  And the fuse implementation doesn't outperform the kernel

implementation in anything at all.  However the kernel one as

available in the kernel source tree doesn't have many write-features,

it can only overwrite files, it cannot create/delete files, etc.  So I

guess if you define "performance" to mean "features" then sure

ntfsmount/ntfs-3g have more features than the public kernel driver.

If you define "performance" to mean "speed" then no ntfsmount/ntfs-3g

can't compare to the kernel except in very limited and meaningless

benchmarks...
btw. even comparing features, the fuse solutions lag behind in some

respects, e.g. no-one can "kill -9" the kernel driver leaving a

corrupt file system on the volume (and under no-one I include the OOM

killer for example!) and another example is that the fuse solutions

require large amounts of ram whereas the kernel driver can happily

function in 1MiB ram and less even as everything is in the page cache

so it will just cause heavy paging whilst the fuse solutions just blow

up / OOM the machine when they find a large directory and the user has

only 32MiB ram for example...  At least I have seen reports of this on

the mailing lists, not that I have ever cared to try.
Best regards,
	Anton

--

Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)

Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK

Linux NTFS maintainer, http://www.linux-ntfs.org/
--

OK, I was exaggerating (notice the smiley).  But I do have a feeling

(and just a feeling, no hard data), that ntfs-3g is making the

in-kernel ntfs filesystem increasingly irrelevant.  And yes, that's

mostly because of the features, but also because the performance is

not at all as bad, as some people would think a userspace filesystem

has to be.
Miklos

--

I think mostly because of dedication of the maintainer (Szabolcs

Szakacsits).  I don't claim fuse has a great part in it, except the

obvious advantages of developing in userspace: no kernel crashes,

easier debugging, etc...
And of course it could be ported back to the kernel (ntfs-3g is GPL

too), and that would probably result in even more speedups.  But

that's not a trivial task either.
Miklos

--

I guess I can keep making this point in various ways until someone

actually notices it:
	This filesystem has only 20 users.
--

At the moment. And that probably exceeds Amiga users, 386 users, some of

the serial port users, several network card users ...
In the past we've merged drivers for network cards where only two

existed in the world. Linus has repeatedly stated he wants to see stuff

people are using getting in. Good clean code that doesn't affect the core

is good reference material.
I think you are (unusually) the one out of step here ?
Alan
--

The reference block filesystem is ext2 (used to be minixfs) - there is no
I appear to be the only one who is looking at the whole picture.
Merging a new filesystem has costs - I don't need to enumerate them.  Do

the benefits of OMFS exceed them?

--

We do not have a stable API for external modules, and part of the deal

is that external modules have the chance of entering the kernel where

they will get API changes automatically.
We are talking about a filesystem even Christoph considers OK.
And who asked about the costs of merging crap like

drivers/infiniband/hw/nes/ ?
Speaking about the latter, with Linus' logic one might argue that OMFS

must not be rejected since it adds support for some hardware...
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

Excatly.  I find it very strange to even consider rejecting a rather

small and very well written driver for let's say "political" reasons.

--

"economic" would be a far more accurate term.
Look, I have repeatedly described the reason why it is probable a poor

tradeoff to merge code such as this.  The only response has been "well

we've done it before", which is largely a non-reason.
You can continue to ignore my logic, but that won't go unnoticed.
Just as a thought exercise: should we merge a small and well-written

driver which has zero users?

--

It seems you missed the first point in my email:
We do not have a stable API for external modules, and part of the deal

is that external modules have the chance of entering the kernel where

they will get API changes automatically.
Plus my other point that one might argue that OMFS adds support for some

hardware in which case a recent commandment by Linus would require it

has to be merged...
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

On Tue, 15 Apr 2008 22:24:32 +0300
That's lawyerly trickery, sorry.  Take some set of guidelines and then say

"you are thereby committed to doing X".
We're not committed to doing anything and it would be bad if we were.

Let's apply common sense and judgement to each case on its own.
--

My favorite gems from the stuff even checkpatch finds in the

INFINIBAND_NES driver, for which Linus has stated explicitely that

merging it in this state in 2.6.25 was correct, can be seen with
  grep -C4 volatile drivers/infiniband/hw/nes/nes_nic.c
When we have the resources to maintain this kind of code, how could a

small filesystem be a problem?
cu

Adrian
-- 
       "Is there not promise of rain?" Ling Tan asked suddenly out

        of the darkness. There had been need of rain for many days.

       "Only a promise," Lao Er said.

                                       Pearl S. Buck - Dragon Seed
--

We've previously merged ones that fit this category and have one user. If

the driver doesn't disrupt core code and is useful to help other people

write drivers then I think we should. It may also get more users as a

result.
This is a free software project not a business enterprise.
Alan
--

On Tue, 15 Apr 2008 19:53:30 +0100
And the same principles apply, don't they?  We seek to minimise overhead

and to use our kernel development hours as effectively as we can.
This is particularly important when person A is thrusting additional work

onto persons B, C, and D.
--

I would disagree as drivers we've merged like that go on to have many

users or give us new developers who stay with the project (and in some
One word if we are getting into the economics of this: Externalities.

There are lots of benefits from merging the code beyond simple value of

code merge including more developers, more reference code, more users.

The ratio of new contributions to maintenance is a different problem and

one I do not think should be conflated with it.
But yes I do think they are different - free software is largely done for

fun, by people who want to contribute. What was it John Betjeman said of

another large volunteer project:
"the result of the independent spirit which still survives in this

country and refuses to be crushed by the money-worshippers, centralizers

and the unimaginative theorists who are doing their best to kill it"
Alan

--

On Tue, 15 Apr 2008, Alan Cox wrote:
	to interject, and in response to a variety of posts in this thread

and in no way drawing solely on yours, alan: despite that it may sound

like some interesting viewpoints are getting aired here, i think the

discussion may've unproductively slewed to a point where andrew's

unfortunately been painted into a corner in which he has little practical

interest.
	it seems like some folks have misinterpreted andrew's remarks as

being variously inflexible, elitist, "political", pro-"corporate-drone",

working at cross-purposes to the spirit of open source, and otherwise.

in rereading the thread, though, he's pretty much been the most

even-handed, moderate, practical, and open to discussion and compromise of

anyone involved.
	he gave an opinion about the merits of OMFS in-kernel vs FUSE,

clarified his stance, remained open to other's comments and criticism,

didn't rise to any bait when people called his ideas "stupid" and whatnot

(despite that he specifically said he had "no strong opinions either

way"), and he compromised: he offered to pull the code into -mm; he said

he'd merge v4.
	known: OMFS has a few bugs (thanks, reviewers), needs testing, and

has outstanding legal questions to resolve.  andrew offered to merge the

next version.
	to ground the topic in terms of practicality: what else would all

of you have andrew do, at this moment, that he hasn't already done?  can

we let him get back to work?
	thanks,
	d

	.

--

Eh you, corporate linux developer, please don't discriminate what's

going in the kernel just because of extra cost. Let linux continue to be

"by the people, for the people" if you see what I mean.
--

By "cost" I refer to extra developer time spent on maintaining the

filesystem.  The most recent example is the write_begin/write_end changes

which took a lot of Nick's time and rather a lot of mine also.
"the people" here are those who work on the kernel.  We want our time to be

spent as effectively as possible.  Sorry if that sounds corporate.

--

My time is definitively spent better on reviewig and if needed fixing a

fun filesystem like omfs that allows linux to deal with fun hardware

instead of fighting the lastet crap code from the corporate drones.

--

That's about the fourth time you've made this assertion.  It is pointless,

it is of no use to anyone and it is unjustly insulting to a large number of

kernel developers.  Many of whom do a hell of a lot more kernel work than

you, I might add.
If you have issues with certain parts of the kernel code then tell us, with

a usable level of detail what they are.  Preferably as a reply-to-all prior

to that code being merged, but after merging is fine too.
Otherwise...

--

No, it isn't. Christoph can be insulting sometimes, but that wasn't it.
Anyone who thinks that Christoph was referring to them obviously

identifies _themselves_ as 'corporate drone who posts crap code'.

Why is it insulting for Christoph to opine that his time is better spent

on reviewing something fun?
I find it extremely annoying when people claim offence on behalf of

unspecified third parties, when no real adult actually _is_ so wet as to
As if Christoph needs encouragement to do just that... :)
--

dwmw2
--

Not so.  There is a habit of treating corporate developers as somehow

inferior to the purer old-timers.  I know - people have told me.  Quite

often.  And for various reasons, those people feel limited in their options

for standing up for themselves.
--

Corporate drones obviously doesn't refer to all of them.  It referes to

those who are primarily corporate and secondarily Linux people of

course.  There's quite a lot of developers who came to Linux with a

corporate background and are fully integrated into the "scene".

Surprising they tend to write mostly good code and are involved with

review and other important activities.

--

I'm not sure I agree. There is a habit of treating people who post crap

code as somehow inferior to the purer old-timers, certainly. And that's

probably not such a good thing, but at least it helps to keep the

signal:noise ratio down a little, overall.
But except to the extent that there is a correlation between 'corporate

developers' and 'people who post crap code', I wouldn't agree with your

statement above. 
We have a lot of people working for large corporations who don't post

crap code, and to whom Christoph almost never promotes an attitude of

violence. Looking at Jon's 'who wrote 2.6.23' analysis, I see quite a
Anyone trying to defend crap code _should_ be limited in their options,

surely? And sometimes, 'corporate developers' do try to defend crap

code, because they've made traditional corporate mistakes like

developing it all in private and presenting it as a fait accomplis

without proper a priori review, and/or haven't budgeted for the

necessary time to fix it.
--

dwmw2
--

Which are only an extra cost for complex filesystem.  Trivial block

filesystems like omfs just use the generic callbacks and can be

converted in a simple sweep.
--

Seconded (as a 95% non-coporate kernel hacker :))
--

I think the exceed them quite easily. The costs are almost nil, while

merging this provides another nice example fs (and one much easier to

follow than ext*) for hardware that does have a few users and will no

doubt get many more
I wasn't aware Linus had introduced a new rule required 500 people sign

up to use a feature before it gets added ?
Alan

--

I'm also very surprised by this, especially as it seems to be applied

very selectively.  This filesystems is an almost 0 maintainance burden

unlike a lot of really crappy driver we're shoving in constantly.

--

Agreed.  And adding to the pile-on, I think that we should be

_encouraging_ rather than discouraging innovative new filesystems.
I rather dislike limiting filesystem maintenance to the "old guard".
See comments like those on Evgeniy's blog:

http://tservice.net.ru/~s0mbre/blog/devel/fs/2008_04_14.html
Sure we do not need to merge every filesystem under the sun, or things

that much better belong in userspace (sshfs), but AFAICS there is not a

huge increase in maintenance cost to add a new kernel filesystem.
	Jeff
--

On Mon, 14 Apr 2008 16:55:04 -0400
Sigh.  I've already addressed that.
A new filesystem such as the one he's working on would, one hopes, be used

by an increasing number of people.  Whereas I expect that the already-tiny

number of people who use OMFS will *fall* over time.
Guys, a few years ago Linus's rule of thumb for merging an FS was,

approximately, "it is already being shipped by distros".  The bar is _much_

lower than that now.  Largely because I personally think that it should be.
--

I thought he wrote in the lead message that it had been picked up

by at least three different distros.  Does that not count?
-sb

--

On Mon, 14 Apr 2008 16:09:41 -0700
Not really - I already agreed to merge v4.

--

Hi.
It is not about pohmelfs, but any other. When did you look at fs/kconfig

last time? There are lots of filesystems which were obsoleted way before

I first time started using computer. Hope you saw with pdp-11 in action

(I only on pictures), since sysv filesystem is very unlikely to get new
And Linus was wrong. This can not happen? He can not change his opinion?

Btw, which distro shipped ext4 before it was included (splitted) into

mainline?
Just say no to double standards!
P.S. Added Linus to copy list, hope it will be interesting...
--

	Evgeniy Polyakov

--

On Tue, 15 Apr 2008 02:32:31 +0400
I've already told you that we no longer apply such a strict test.
--

Thanks to Bob Copeland for taking the time to submit this for mainline.

Please don't mistake the resulting debate as a sign we don't appreciate the

effort of making it available and getting it reviewed.
This zero maintenance burden idea is a little suprising to me.  The kernel

interfaces do change and we either end up changing every filesystem to

support the new interface or we need to carry around the old ones for the old

filesystems.
Even though OMFS seems to be using the generic interfaces well, there is still

a testing burden for every change.  Someone needs to try it, report any

problems and get them fixed.  Since none of the people making the changes is

likely to have an OMFS test bed, all of that burden will fall on Bob, his

users, and anyone who tries to compile the module (Andrew).
Unlike all the device drivers we don't want floating around out of the tree,

filesystem authors do have a choice between FUSE and being in-kernel.  Since

OMFS has been maintained out of tree since 2.6.12 or so, Bob probably has a

very good idea of how much time he has needed to spend updating things for

each release.
So, there are two big questions I'd ask:
* Which setup (kernel or fuse) would Bob find easier to maintain things?

* Is this a first step toward more development and an increased user base?
Tossing it in -mm isn't a bad step toward getting attention to see if more

people want to use it.
-chris

--

switching to FUSE also has a cost for users, namely that they need to have

FUSE setup (and the various interactions and deadlocks that can happen

with a userspace filesystem, such as swapping to it)
as a user I would prefer to see filesystems (even ones I don't expect to

uer) be all treated the same way, not have to figure out that to use this

list of filesystems I configure them in the kernel, and to use that list

of filesystem I have to run FUSE.
for testing, or for things that aren't really filesystems (views into

version control systems, tarballs, etc) FUSE is a good match.
but for real filesystems it's a poor second.
David Lang
--

Swap on loopback could deadlock with whatever file system, not only FUSE.
The correct and safe way to use swap with FUSE is using the fuseblk fstype

and a swap file on the FUSE file system (so it's exactly the same as one

would do with an in-kernel file system). This completely avoids the usage

of user space during swapping. This is in use for over a year with ntfs-3g,

e.g. by Knoppix and Mr Klaus Knopper wrote only positive things to me so

far.
	Szaka
--

NTFS-3G:  http://ntfs-3g.org

--

The recent deprecation of iget was probably the most involved of any.  It

wasn't that bad, but the #ifdef jungle going back to at least .17 is getting

somewhat hairy (I have 19 of them).  I would say over the past year or so

most of the changes have been <10 liners with kmem_cache_* interface changes.

Luckily through review here, that got zapped anyway.
In the normal case, I'll grab an -rc3 kernel or so, try to compile it, if

it fails, look at minix to see the minimum set of changes I need to make.

The challenge for me maintaining it out-of-tree is that sometimes the git

logs do not carry enough context to know why XYZ changed and if there's a
Well, FUSE is easier from the standpoint of having somewhat more control.

On the other hand, that also results in an inferior product if it isn't given

the many-eyes treatment.  For the code itself, it's really not that much

different since FUSE is so close to the VFS API anyway.
For your second question:
More users are always welcome.  At present, the FUSE version implements a few

things that the kernel module does not - better/different block allocation

algorithm, growing truncate, automatic byte-swapping for the ReplayTV model

that had a busted block layer.  However, the FUSE version is known to be buggy

with respect to every day use (I had one MD5 fail among 20G of files in one

copy test).
I could see adding 1 and 2 to the kernel version, but probably not the

byte-swapping bit.  I'm not sure what to do from there on.
As a user, I use the kernel module because it works, it's faster, and the

other bells and whistles aren't that important to me.  It's hard to tell

what the user community prefers, because the kernel module was first and

so far no one has had a reason to try the FUSE implementation (if they

have, I don't know about it).
--

Bob Copeland %% www.bobcopeland.com

--

What does the replaytv byteswap - the entire blocks or some fields ?

--

Entire blocks (both meta- and normal data).  E.g. the filename 'Filename'

would look like 'eliFeman' on disk.  I have a dm target that can do this

gathering dust somewhere.
--

Bob Copeland %% www.bobcopeland.com

--

On Mon, 14 Apr 2008 13:18:53 -0400
Thats what I was wondering. The dm target would be useful and is also

needed for certain old 68K boxes with libata

--

He doesn't need to.
We'll frequently add features in the expectation that they will be useful

to a significant part of our user base.  It's a part of the decision-making

process.
Many features will have a small number of users initialy, but that grows

over time.  otoh, OMFS's user base is small and it appears that it will

dwindle.
(And in the past Linus has set the bar *much* higher for new filesystems

than I have..)

--

Well, 20 may have been aiming just a tad low.  I'd never make it in sales.

Here are the stats from linux-karma.sf.net:
 - most recent release, for 2.6.25: 36 downloads since 3/16/08

 - # subscribers on mailing list: 39 [1]

 - most # of downloads of a given release: 252 (May-Oct 2007) [2]
[1] Some subscribers may not use the FS at all since there is also

software for using the ethernet interface.  Both are discussed on

the list.
[2] I doubt that 252 d/ls translates to that many _current_ users; a

year is a long time in consumer electronics.  I don't have stats on

unique IPs.
I'm unaware of ReplayTV users - one guy contacted me once and then
You guys would know best.  I can see both arguments...
--

Bob Copeland %% www.bobcopeland.com 
--

Well as I say - no strong opinions either way here.  It just seems a bit

odd to burden the kernel with an additional fs for such a small user base

when either options exist.
OTOH when we merge a new fs that also has the side-effect of increasing the

active developer base - the maintainers of the fs fix other stuff.  And

I'll do anything to get some acpi bugs fixed ;)
So.. whatever.  I'll keep a look out for v4.

--

Roger that, thanks.
--

Bob Copeland %% www.bobcopeland.com 
--