login
Search
Jörn Engel announced LogFS, "a scalable flash filesystem." The project's home page notes that LogFS aims to be the successor of JFFS2, "the two main problems of JFFS2 are memory consumption and mount time. Unlike most filesystems, there is no tree structure of any sorts on the medium, so the complete medium needs to be scanned at mount time and a tree structure kept in-memory while the filesystem is mounted. With bigger devices, both mount time and memory consumption increase linearly. JFFS2 has recently gained summary support, which helps reduce mount time by a constant factor. Linear scalability remains. YAFFS also appears to be better by a constant factor, yet still scales linearly."
In contrast, Jörn compared his new LogFS, "LogFS has an on-medium tree, fairly similar to Ext2 in structure, so mount times are O(1). In absolute terms, the OLPC system has mount times of ~3.3s for JFFS2 and ~60ms for LogFS." Regarding its stability, he noted, "LogFS works and survives my testcases. It has fairly good chances of not eating your data during regular operation. There are still two known bugs that will eat data if the filesystem is uncleanly unmounted. Also still missing is wear leveling." Thomas Gleixner reviewed the code and offered the following summary, suggesting the code has a ways to go before it replaces JFFS2, "the code is far from being useful on real world hardware. The error handling via BUG() is just making it useless. Also please fix the coding style and other issues from the seperate review. Some useful comments would make a functional review way easier."
From: Jörn Engel [email blocked] To: Andrew Morton [email blocked] Subject: [PATCH 0/2] LogFS take two Date: Mon, 7 May 2007 23:59:14 +0200 Motivation: Linux currently has 1-2 flash filesystems to choose from, JFFS2 and YAFFS. The latter has never made a serious attempt of kernel integration, which may disqualify it to some. The two main problems of JFFS2 are memory consumption and mount time. Unlike most filesystems, there is no tree structure of any sorts on the medium, so the complete medium needs to be scanned at mount time and a tree structure kept in-memory while the filesystem is mounted. With bigger devices, both mount time and memory consumption increase linearly. JFFS2 has recently gained summary support, which helps reduce mount time by a constant factor. Linear scalability remains. YAFFS also appears to be better by a constant factor, yet still scales linearly. LogFS has an on-medium tree, fairly similar to Ext2 in structure, so mount times are O(1). In absolute terms, the OLPC system has mount times of ~3.3s for JFFS2 and ~60ms for LogFS. Motivation 2: Flash is becoming increasingly common in standard PC hardware. Nearly a dozen different manufacturers have announced Solid State Disks (SSDs), the OLPC and the Intel Classmate no longer contain hard disks and ASUS announced a flash-only Laptop series for regular consumers. And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards, etc. Flash behaves significantly different to hard disks. In order to use flash, the current standard practice is to add an emulation layer and an old-fashioned hard disk filesystem. As can be expected, this is eating up some of the benefits flash can offer over hard disks. In principle it is possible to achieve better performance with a flash filesystem than with the current emulated approach. In practice our current flash filesystems are not even near that theoretical goal. LogFS in its current state is already closer. Current state: LogFS works and survives my testcases. It has fairly good chances of not eating your data during regular operation. There are still two known bugs that will eat data if the filesystem is uncleanly unmounted. Also still missing is wear leveling. Handling of read/write/erase errors currently is BUG(). It is on my list, no need to remind me. :) Overall I consider this to be -mm material. It would be good to get some review and have the usual allyesconfig crowd build it and find coverity bugs and the like. http://logfs.org/logfs/ may have some further information. Shameless plug: I have quit my job last November to concentrate on LogFS. While I have found one sponsor kind enough to fund me, my monetary reserves are fairly stressed. Fairly soon I will be forced to take an old-fashioned job again and work on other less exciting stuff. So if anyone needs a fast flash filesystem and has spare money to spend, please contact me. Jörn -- Everything should be made as simple as possible, but not simpler. -- Albert Einstein
From: Thomas Gleixner [email blocked] Subject: Re: [PATCH 0/2] LogFS take two Date: Tue, 08 May 2007 09:39:37 +0200 On Mon, 2007-05-07 at 23:59 +0200, Jörn Engel wrote: > LogFS has an on-medium tree, fairly similar to Ext2 in structure, so > mount times are O(1). In absolute terms, the OLPC system has mount > times of ~3.3s for JFFS2 and ~60ms for LogFS. Impressive number > Motivation 2: > > Flash is becoming increasingly common in standard PC hardware. Nearly > a dozen different manufacturers have announced Solid State Disks > (SSDs), the OLPC and the Intel Classmate no longer contain hard disks > and ASUS announced a flash-only Laptop series for regular consumers. With a hardware controller which allows no direct access to the flash. > And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards, > etc. which again do not allow direct access to the flash > Flash behaves significantly different to hard disks. In order to use > flash, the current standard practice is to add an emulation layer and > an old-fashioned hard disk filesystem. As can be expected, this is > eating up some of the benefits flash can offer over hard disks. > > In principle it is possible to achieve better performance with a flash > filesystem than with the current emulated approach. Err, where does JFFS2 use a block emulation layer ? > Current state: > > LogFS works and survives my testcases. It has fairly good chances of > not eating your data during regular operation. There are still two > known bugs that will eat data if the filesystem is uncleanly > unmounted. Also still missing is wear leveling. Are you going to make logfs play with UBI ? > Handling of read/write/erase errors currently is BUG(). It is on my > list, no need to remind me. :) > > Overall I consider this to be -mm material. I don't. It seems fs developers tend to have their own view of how to get stuff mainline. The code is far from being useful on real world hardware. The error handling via BUG() is just making it useless. Also please fix the coding style and other issues from the seperate review. Some useful comments would make a functional review way easier. > It would be good to get > some review and have the usual allyesconfig crowd build it make allyesconfig does not work for you ? tglx
From: Jörn Engel [email blocked] Subject: Re: [PATCH 0/2] LogFS take two Date: Tue, 8 May 2007 13:41:30 +0200 On Tue, 8 May 2007 09:39:37 +0200, Thomas Gleixner wrote: > > > Motivation 2: > > > > Flash is becoming increasingly common in standard PC hardware. Nearly > > a dozen different manufacturers have announced Solid State Disks > > (SSDs), the OLPC and the Intel Classmate no longer contain hard disks > > and ASUS announced a flash-only Laptop series for regular consumers. > > With a hardware controller which allows no direct access to the flash. > > > And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards, > > etc. > > which again do not allow direct access to the flash I know that and I have talked to manufacturers. Not allowing direct access is common practice today, but I didn't encounter much opposition against allowing it in the future. What appears to be holding them back is that there would be absolutely no value in it right now. With direct flash access, which filesystem should users choose for their 32GB SSD? > > Flash behaves significantly different to hard disks. In order to use > > flash, the current standard practice is to add an emulation layer and > > an old-fashioned hard disk filesystem. As can be expected, this is > > eating up some of the benefits flash can offer over hard disks. > > > > In principle it is possible to achieve better performance with a flash > > filesystem than with the current emulated approach. > > Err, where does JFFS2 use a block emulation layer ? It doesn't. Motivation 2 is about SSDs, USB sticks, SD-Cards, etc. JFFS2 is motivation 1. > Are you going to make logfs play with UBI ? It is not very high on my priority list. > > Handling of read/write/erase errors currently is BUG(). It is on my > > list, no need to remind me. :) > > > > Overall I consider this to be -mm material. > > I don't. It seems fs developers tend to have their own view of how to > get stuff mainline. Maybe. My view is that I have to solve any problems found until people consider the code good enough by whatever metric. The final criterium appears to be quite fuzzy. > The code is far from being useful on real world hardware. The error > handling via BUG() is just making it useless. On NOR hardware? How many write/erase failures does one commonly encounter there? Those things will need to get sorted, sure. But I doubt whether LogFS is useless on _all_ hardware because of this. > Also please fix the coding style and other issues from the seperate > review. Sure. > Some useful comments would make a functional review way easier. Common problem. Implementor doesn't know what comments would be useful and reviewer doesn't know where to start without useful comments. I will try to add some and would love to see suggestions. > > It would be good to get > > some review and have the usual allyesconfig crowd build it > > make allyesconfig does not work for you ? It does. But I don't have a coverity license, just to give one example. Jörn -- The wise man seeks everything in himself; the ignorant man tries to get everything from somebody else. -- unknown
Flash v. Traditional medium
Does LogFS solves the problem which JFFS2 tries to solve?
The main problems with normal file systems is that they have "hot spots" - places which are updates very very frequently. While this pose no problem to normal hard drives, flashes have limited number of write cycles. If you would write too often into single disk location - e.g. root direntry - there are very real chances that flash would break.
Structure of JFFS2 avoids having any "hot spots" by having tree in memory and writing "journal" - changes - in rotating manner, equally utilizing (and wearing) flash space. I hardly see any on-disk tree-based structure which can avoid having such "hot spots" - you would have to have often written place containing tree root (or pointer to rotating tree root).
Does LogFS solves the
Does LogFS solves the problem which JFFS2 tries to solve?
Not now:
"Also still missing is wear leveling."
Is wear leveling still needed?
If memory serves, some flash hardware have themselves some kind of wear leveling integrated, so in these cases, it would be unneeded additional complexity..
Yes, you are write. No, you
Yes, you are write.
No, you are wrong.
Your both.
Depends on what devices your talking about.
Direct-to-flash devices are called "MTD devices". Harddrives are called "Block devices", and things like your keyboard card called "Char devices".
USB devices, and MMC devices and other such 'normal' consumer flash devices have extra hardware in them that emulate block devices. Part of what that hardware does is do the load leveling and keep track of 'hotspots' in the flash.
For those sort of things you have to use a traditional file system like ext2 or whatnot.
Now with MTD devices the OS is allowed direct access to the flash. With those the smarts need to be in the software.
Obviously it's desirable to have MTD, if you have the software to support it.
Unfortunately due to certain inflexible proprietary operating systems us mortals will have to be content with our run of the mill consumer devices.
(I know this is quite a bit after the question was asked, but other people will stumble across this article and ask the same questions)
Wear leveling was not an
Wear leveling was not an afterthought but planned from the beginning. The most important part is already there: LogFS stores the erase count for all blocks. JFFS2 only does stochastic wear leveling. Sometimes it picks a random block instead of the most empty one. There are voices claiming that the JFFS2 approach does not work too well.
Of course LogFS still has to use the erase counts. It is not fully mature yet. But infrastructure is there and adding wear leveling does not require a format change.
uggh
that was rough. I now, that is how the lkml works, but if the developer runs away with his fs, i wouldn't blame him. I was ready for swears of any kind at the end of the email.
lkml can be rough, but
lkml can be rough, but rightfully so most of the time. In this case the developer was not following style guidelines and makes a number of rash/uninformed statements that they were called on. If I were a lkml member I would also be confused as to where this would "fit" and what problem this solves.
Most flash drives available today are not accessed directly and have "wear leveling" built into the chip sets. So while it's a good idea to have a low-write file system I don't know if LogFS is the best solution for that case. Wouldn't extending jbd and modifications to ext2/3 be the better path?
If you are talking directly to flash wear leveling is essential and is missing from this filesystem. This is why jffs will continue to dominate this segment of the market.
While there are still bits
While there are still bits missing to implement wear leveling in logfs, the file system was designed from the start with exactly this in mind, the basic premise being "never overwrite a block in place", it just isn't fully implemented yet. It is however not possible to retrofit this concept into traditional file systems like ext2 that is made to overcome the problems of spinning disks.
OTOH, the days of jffs2 (jffs has recently been removed) are numbered, because of the inherent scalability problems arising from the need to read all metadata into RAM at mount time.
Of course I ran away with my
Of course I ran away with my filesystem. I ran back to my desk to do my homework. Most of the review comment were just, after all. ;)
LogFS name
Could you name it something else? There are at least 20 log-structured filesystems by now all called LogFS.
But this one isn't log
But this one isn't log structured, it's journal based, unlike jffs2 which is log structured ;-)
That was last year. Then
That was last year. Then Arnd noticed that my filesystem scales logarithmically. So obviously it must be called LogFS, the only rational choice. ;)
Actually, LogFS also is a log-structured filesystem. I just like to ignore that fact when making my JFFS2 joke and call it a journaling filesystem, as it also has a journal.
Name is Wierd
Why not just call it FlashFS, sounds nice to me...
I agree! Call it FlashFS
I didn't get the "log" thing until the scaling link was mentioned. I've studied engineering mathematics to a high level. The average consumer has no chance!
Hell! Doesn't your own sig say to make things as simple as possible, but no simpler?
I know we all like our little in-jokes, but then, notice what basically every fix for windows is, for example. "Oh, you just need to change *insert random meaningless key* in the registry."
It's just meaningless "magic". I'm sure it made perfect sense to whoever wrote that software at the time.
Sure it fixes the problem, but learning that fix doesn't teach you how to fix anything else.
Obscurity is the problem, and excessively smart-alec names don't help either, even if they are "traditional".
Start with naming things for what their function is. -> Filesystem on Flash. FlashFs.
Never obfuscate with "Inner club" knowledge when you could make something intuitive and transparent.
Being the only one who knows the super-secret-special-spell doesn't make you any smarter, it just makes everyone else dumber. IMHO this was what Einstein was trying to say there.
Wizard's First Rule: People Are Stupid.
First corollary: a people == n*Individual. (n is a natural number).
You are an individual.
Ergo, no matter how much knowledge you accrue, you will always be stupid too.
To extrapolate from this - if you want to know what you should be writing in your comments, just consider yourself to be explaining your code to a very bright, yet inexperienced child.
You'll find such comments immeasurably helpful when you need them.
From the sounds of it, and as it is open sourced, FlashFS probably will become *the* way to do a filesystem on Flash.
Problem solved.