login
Search
I hope I got the CC list right. Apologies to anyone in didn't include and anyone I shouldn't have included. The basic idea is to include an idea from VMS that seems to be quite useful: version numbers for files. The idea is that whenever you modify a file the system saves it to na new copy leaving the old file intact. This could be a great advantage from many view points: 1) it would be much easier to do package management as the old version would be automatically saved for a package management system to deal with. 2) backups would also be easier as all versions of a file are automatically saved so it could be potentially very useful for a company or the like. There are probably many others but these were the two that I liked best. Revision numbers could be specified as follows: /path/to/file:revision_number I think that this can be done without breaking userspace if the default was to open the highest revision file if no revision number is specified. The userspace tools would need to be updated to take full advantage of the new system but if the delimiter between the path and revision number were chosen sensibly then the changes to most of userspace would be minimal to non-existant. Personally, I think that the bulk of the implementation could be in the core fs code and the modifications to individual filesystems would be minimal. The main implementation ideas I have (however, I am no kernel expert =) are adding an extra field to struct file and struct inode called int revision (as version is already taken) that would hold the number of the file revision being accessed. Another problem could be the increased usage of disk space. However if only deltas from the first version were stored then this could cut down on space, or if this were too slow to open a file then the deltas could be off every tenth revision (ie 0,10,20,30... where 0,10,20... are full copies of the file). There would need to be a tool of some describtion to remove old revisions ...
This is one of those things that seems like a good idea, but frequently ends up short. Part of the problem is that "whenever you modify a file" is ill-defined, or rather, if you were to take the literal meaning of it you'd end up with an unmanageable number of revisions. Furthermore, it turns out that often relationships between files are more important. Thus, in the end it turns out that this stuff is better handled by explicit version-control systems (which require explicit operations to manage revisions) and atomic snapshots (for backup.) -hpa -
And there are files that are not important at all.
Would you save every temp file? To be meaningful you would need to know
ZFS is the cool new thing in that space. Too bad the license makes it
hard to incorporate it into the kernel. (I am one of those people that
believe that Linux should support EVERY file system, no matter how old or
obscure.)
--
"ANSI C says access to the padding fields of a struct is undefined.
ANSI C also says that struct assignment is a memcpy. Therefore struct
assignment in ANSI C is a violation of ANSI C..."
- Alan Cox
-
Do you have example discs that can be mounted to test it? If you do, I
will consider doing it.
I have a couple of older DOS filesystems that got dropped out years ago
that I actually need to mount disks that i may rewrite for 2.6.x.
Now all i need is the time.
And speaking of obscure information...
I have a bunch of PCMCIA spec documents from the PCMCIA standards
association from the late 90s. Would anyone involved in maintaining the
PCMCIA code be interested in it? (Especially if they are in Portland.)
It has been a while since I have even needed to look at it and I hate for
it to go to waste if it can be of any use. (Bit late now, I know...)
--
"ANSI C says access to the padding fields of a struct is undefined.
ANSI C also says that struct assignment is a memcpy. Therefore struct
assignment in ANSI C is a violation of ANSI C..."
- Alan Cox
-
One of the key points of the implementation would have to be the ability to delete old revisions without affecting the subsequent revisions. This would allow people to keep the number of revisions down. Also if each revision is in effect a patch on the last revision it could cut down the disk space required to store them, or if that takes to long to read a file then have every tenth version (0,10,20,30...not the tenth versions I know but easier to read) as a full version of the file which I hadn't considered that but I did think that you could remove the old revisions of a file at some configurable time after. This would allow recovery in case of accidental deletion but should keep the disk space Possibly but if I use it to manage my entire system (ie as a package manager) then the system would likely explode if I tryed to update or remove a key package whilst the system was running. With the kernel -
This already exists -- it just not open sourced, and you could spend years trying to create it. Trust me, once you start dealing with the distributed issues with this, its gets very complex. I am not meaning to discourage you, but there are patents already filed on this on Linux. So you need to consider these as well, and there are several folks who are already doing this or have done it. If it goes into Microsoft endorsed cross licensed Linuxes It may be ok (Vertias sold this capability to Microsoft already, about 12 patents there to worry over). There's also another patent filed as well. It's a noble effort to do a free version, but be aware there's some big guns with patents out there already, not to mention doing this is complex beyond belief. Anyway good luck. ~~~~ -
I reviewed your sample implementation, and it appears to infringe 3 patents already. You should do some research on this. ~~~~ -
Are you able to tell us which areas of the code infringe existing patents? Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! -
Yes. -
First of all, you are responding to someone in the UK, I thought they didn't even have software patents over there. Second, I didn't see any implementation, just a high level description. Finally advising anyone (who is not an actual patent lawyer that could correctly interpret the language and scope of a patent) to go search out patents seems pretty bad advice. That can only result in not even attempting to research some potentially new and innovative approach. Researching prior published work in the area is considerably more helpful. Especially when something is complex beyond belief it has probably attracted various researchers over time and there are most likely various different solutions that have been explored previously. Such existing work can form a good basis for further work. Finally, even if there are patents they could be too limited in scope, overly broad, can be invalidated due to prior art. It may also be possible that a patent holder has no problem granting a royalty free license for a GPL licensed implementation. Jan -
When you get into the recycling issues with storage, the patents come into play. Also, using the file name to reference revisions is already the subject of a patent previously filed (I no longer own the patent, I sold them to Canopy). There is a third one about to be issued. The patents are: * 6,862,609 **6,795,895 and this one about to be issued: http://www.wipo.int/pctdb/en/fetch.jsp?LANG=ENG&DBSELECT=PCT&SERVER_TYPE=19&am... The last one was filed with WIPO and has international protection, UK included. Jeff * Jeff -
I have no idea about patents so if anyone could point me in the right direction I would be most obliged Jack -
Thanks for clarifying Alan, I was uncertain about current patent laws in the UK and abroad. I know this area has undergone considerable debate and changes recently, and I have not been keeping up with all of it. -
Congratulations on obtaining those patents, I hope they will be used
wisely. I am however not a patent lawyer and as such in no position to
evaluate their claims.
As a more useful response, the original poster may want to look at some
of the prior work in this area, I just picked a couple,
(Cedar File System from Xerox PARC)
A Caching File System for a Programmer's Workstation (1985)
Michael D. Schroeder, David K. Gifford, Roger M. Needham
(Vax/VMS System Software Handbook)
(TOPS-20 User's Manual)
(Plan 9 (file system))
Plan 9 from Bell Labs (1990)
Rob Pike, Dave Presotto, Sean Dorward, Bob Flandrena, Ken Thompson,
Howard Trickey, Phil Winterbottom
(Elephant File System)
Elephant: The File System that Never Forgets (1999)
Douglas J. Santry, Michael J. Feeley, Norman C. Hutchinson
Workshop on Hot Topics in Operating Systems
Deciding when to forget in the Elephant file system (1999)
Douglas S. Santry, Michael J. Feeley, Norman C. Hutchinson,
Alistair C. Veitch, Ross W. Carton, Jacob Otir
(Ext3Cow)
Ext3cow: The Design, Implementation, and Analysis of Metadata for a
Time-Shifting File System (2003)
Zachary N. J. Peterson, Randal C. Burns
Sites like portal.acm.org and citeseer.ist.psu.edu are good places to
find copies of these papers. They also provide links to other work that
either is cited by, or cites these papers which is a convenient way to
find other papers in this area.
Researching, designing and implementing such systems is a lot of fun,
admittedly often more fun than long term debugging and/or maintaining,
but that is life. Don't get too upset if the end result cannot be
included in the main kernel. Just start over from scratch, you may just
end up with an even better design the second time around.
Jan
-
Thank you very much for the info and the advice. I would also like to thank everyone for the help and enchouragement that they have given to me. Jack -
Also Files/11 Basic versioning goes back to at least ITS Not sure how old doing file versioning and hiding it away with a tool to go rescue the stuff you blew away by mistake is, but Novell Netware 3 certainly did a good job on that one -
The trick in the NetWare 3 model was to segregate the directory entries onto special reserved 4K directory blocks (128 byte dir records). When it came time to purge storage after the file system filled, an entire 4K block and all chains was deleted during block allocation for new files. The dir blocks were ordered by date -- oldest ones got purged first. The model worked very well until compression was added to the filesystem, then it started getting complex. I would be willing to help instrument the NetWare 3 model in this proposal on ext3, since this is a basic versioning model and would provide coverage for 99% of the folks needing this capability. I am available for questions. -
I resigned as Chief Scientist of Solera Networks May 8 this year -- mostly because I was not allowed to have any freedom, including working on free Linux projects. This went on for almost 4 years (since 2003). Now that I am independent again, I can work on stuff like this again. I started a new company with John Noorda (Ray Noorda's oldest son) who runs Canopy now. That's my current status. I am an owner and entrepeneur again. So I have a lot of free time and will from now on. Jeff -
DEC had versioning files systems 30 years ago. Any patents on their style must certainly have expired long ago. Look at RSX-11 and other seventies era operating systems. This is ancient stuff. -
Would such patents still be valid? He does not seem to be describing anything that the ICL VME/B operating system did not do in the 1970s, so any applicable patents should have eithe expired by now or be invalidated by prior art. -
It may be the coolest, but there are others as well. Btrfs looks good, nilfs finally has a cleaner and may be worth a try, logfs will get snapshots sooner or later. Heck, even my crusty old cowlinks can be viewed as snapshots. If one has spare cycles to waste, working on one of those makes more sense than implementing file versioning. Jörn -- "Security vulnerabilities are here to stay." -- Scott Culp, Manager of the Microsoft Security Response Center, 2001 -
Too bad everyone is spending time on 10 similar-but-slightly-different filesystems. This will likely end up with a bunch of filesystems that implement some easy subset of features, but will not get polished for users or have a full set of features implemented (e.g. ACL, quota, fsck, etc). While I don't think there is a single answer to every question, it does seem that the number of filesystem projects has climbed lately. Maybe there should be a BOF at OLS to merge these filesystem projects (btrfs, chunkfs, tilefs, logfs, etc) into a single project with multiple people working on getting it solid, scalable (parallel readers/writers on lots of CPUs), robust (checksums, failure localization), recoverable, etc. I thought Val's FS summits were designed to get developers to collaborate, but it seems everyone has gone back to their corners to work on their own filesystem? Working on getting hooks into DM/MD so that the filesystem and RAID layers can move beyond "ignorance is bliss" when talking to each other would be great. Not rebuilding empty parts of the fs, limit parity resync to parts of the fs that were in the previous transaction, use fs-supplied checksums to verify on-disk data is correct, use RAID geometry when doing allocations, etc. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. -
How many of these feature could be (at least partially) implemented in the core code so that it was easier to port filesystems to use them. Certainly versioning should be able to be implemented, possible without any FS support at all, in the core code. Centralising the implementations could allow a great deal of FS custiomisation/improvements with very little new per FS code. Jack -
There definitely seems to be an inflation of new filesystems. These days all the cool kids either write their own virtualization layer or their own filesystem. No idea why that happened, two years ago Consider me sceptical. Here is my personal opinion when looking at the list: Chunkfs, tilefs - research projects. At this moment nobody knows whether either of the approaches works or not. Once that is proven, the tricks should get incorporated in existing filesystems. Dave Chinner seems to be working on similar stuff for XFS already. Assuming he can deliver, a chunked/tiled/... XFS is useful while chunkfs and tilefs are mostly educational. It doesn't have to be XFS, Ext4 would do just as well. Logfs - flash filesystem. Btrfs - disk filesystem. Disk optimization comes down to avoiding seeks like the plague. Flash requires garbage collection, wear leveling and avoiding writes like the plague. It is quite unlikely that either filesystem will do well in the other's domain. At least some amount of code will remain seperate. Subsets of code might be useful for both. Collaboration on that level would be useful. Jörn -- Joern's library part 4: http://www.paulgraham.com/spam.html -
I view some of the attempts for "from scratch" filesystems as ways of testing out various designs as "proof-of-concepts". It's a great way of demo'ing ones ideas, to see how well they work. There is a huge chasm between a proof-of-concept and a full production filesystem that has great repair/recovery tools, etc. That's why it's so important to do the POC implementation first, so folks can see how well it works before investing a huge amount of effort to make it be production-ready. So I actually think the number of these new filesystem proposals are *good* things. It means people are interested in creating new filesystems, and that's all good. Eventually, we'll need to decide which design ideas should be combined, and that may be a little tough to the egos involved, but that's all part of the darwinian kernel programming model. Not all implementations make it into the kernel mainline. That doesn't mean that the work that was done on the various schedular proposals were useless; they just helped demonstrate concepts and advanced the debate. Regards, - Ted -
I just wish that people would learn from the mistakes of others. The
MacOS is a prime example of why you do not want to use a forked
filesystem, yet some people still seem to think it is a good idea.
(Forked filesystems tend to be fragile and do not play well with
I would like to see more clarification from the designers as to what
problem they are trying to solve. Some of the goals seem to be laudable,
but some are not problems that I worry about.
I see filesystems that are trying to handle the flakeyness of hardware.
That is useful to me. I also see people who are trying to archive every
little change for "legal reasons". I have a hard time with this one
because I have a hard enough time keeping spare hard drive space for the
stuff I want, not the space that someone else wants me to keep.
What I really want are high throughput systems where I can write and read
as fast as the hardware will allow. (And then I want faster hardware.)
--
"ANSI C says access to the padding fields of a struct is undefined.
ANSI C also says that struct assignment is a memcpy. Therefore struct
assignment in ANSI C is a violation of ANSI C..."
- Alan Cox
-
Jeremy Alison used to be the one who was always pestering me to add Streams support into ext4, but recently he's admitted that I was right that it was a Very Bad Idea. As I mentioned in my Linux.conf.au presentation a year and a half ago, the main use of Streams in Windows to date has been for system crackers to hide trojan horse code and rootkits so that system administrators couldn't find them. :-) - Ted -
But... that's an essential feature in Windows! Otherwise, how would Microsoft produce a RKDK (Rootkit development kit) to sell to Sony? -hpa -
Well, I just want to make sure people know that Samba isn't asking for it any more, and I don't know of any current requests outstanding from any of the userspace projects. So there's no one we need to ship off to the re-education camps about why filesystem fork/streams are a bad idea. :-) - Ted -
NFSv4 also has support for "named attributes"/forks/streams. As far as I can recall, though, the only requests I've seen for it (aside from people just running through NFSv4 feature checklists) have been from people that really just wanted extended attributes. And it'd probably actually be pretty easy to extend the v4 protocol to support extended attributes if someone wanted to. --b. -
(sorry for the late reply, just got back from holiday) The only valid use of Streams in Windows I've seen was a virus checker that stored a hash of the file in a separate stream. Checking a file was a matter of rehashing it and comparing against the hash stored in the special hash data stream for that particular file. Erik --=20 They're all fools. Don't worry. Darwin may be slow, but he'll eventually get them. -- Matthew Lammers in alt.sysadmin.recovery
And even that's not a valid use. All the virus would have to do is to infect the file, and then update the "special hash data stream". Why is it that when programmers are told about streams as a potential technology choice, it makes their thinking become fuzzy? :-) - Ted -
I meant valid like "not used as malware". I agree a virus could recompute the hash and go unnoticed. Erik --=20 They're all fools. Don't worry. Darwin may be slow, but he'll eventually get them. -- Matthew Lammers in alt.sysadmin.recovery
>>>>> "Erik" == Erik Mouw <mouw@nl.linux.org> writes: Erik> (sorry for the late reply, just got back from holiday) Erik> The only valid use of Streams in Windows I've seen was a virus Erik> checker that stored a hash of the file in a separate Erik> stream. Checking a file was a matter of rehashing it and Erik> comparing against the hash stored in the special hash data Erik> stream for that particular file. So what was stopping a virus from infecting a file, re-computing the hash and pushing the new hash into the stream? You need to keep the computed hashes on Read-Only media for true security, once you let the system change them, then you're toast.... John -
On Thu, 5 Jul 2007 09:57:40 -0400 I'm not a huge fan of streams, but I'm pretty sure there are various encryption tools that let us verify and validate the source of data. It's entirely possible the virus checker wasn't doing it right, but storing verification info in an EA or stream isn't entirely invalid. You still need an external key that you do trust of course. -chris -
Nothing, but the same holds for virus checkers that store the hash in a separate file. The only advantage of storing the hash in a stream is Agreed. Erik --=20 They're all fools. Don't worry. Darwin may be slow, but he'll eventually get them. -- Matthew Lammers in alt.sysadmin.recovery
Unfortunately, I can't do OLS this year, but anyone who wants to talk on these things can drop me a line and we can setup phone calls or whatever for planning. Adding polish to any FS is not a one man show, and so I know I'll need to get more people on board to really finish btrfs off. One of my long term goals for btrfs is to figure out the features and layout people are most interested in for filesystems that don't have to be ext* backwards compatible. I've got a pretty good start, but I'm Definitely. There's a lot of work in the DM integration bits that are not FS specific. -chris -
I have no filesystem programming experience, but I am certainly interested, and I'm spending some time reading through the code that you've written so far. Oh, and running it - though I'm probably going to want to fiddle with some smaller filesystems than my entire Maildir set if I want to make any sense of the structure dumps! That and of course if I get involved in development I can be sure that my favourite workload (big Cyrus installs) is well optimized for! Actually, my biggest interest is decent unlink performance, in particular when you are unlinking multiple items in a directory or even the entire directory plus everything in it. I find that to be an incredibly slow and IO hurting operation. We run cyr_expire (the process in Cyrus that actually deletes expunged messages) once per week, and only one process at a time on a machine which might have 20 otherwise busy instances of Cyrus running - because the IO hit on those data partitions is massive. Load average more than doubles and the log entries for commands which took longer than a second to return increase massively. And this is on a Sunday when there's barely any use compared to a weekday. So yeah, my main interest is making unlink (especially multiple unlinks from the same directory) into a less extreme experience. Bron. -
Interesting that you mention the multitude of file systems because
I was very surprised to see NILFS being promoted in the latest Linux
Magazine but no mention of the other more important file systems
currently in work like UnionFS ChunkFS or ext4 so publisized.
I can say I was disapointed of the article. I still didn't
see any real prove that NILFS is the best file system since bread.
Neither I see any comments on nilfs from Andrew and others and
yet this is the best new file system coming to Linux. Maybe I missed
something that happened in Ottawa.
/Sorin
On Mon, 18 Jun 2007 05:45:24 -0400, Andreas Dilger <adilger@clusterfs.com>
--
Best Regards
Sorin Faibish
Senior Technologist
Senior Consulting Software Engineer Network Storage Group
EMC²
where information lives
Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : sfaibish@emc.com
-
The underlying internal implementation of something like this wouldn't be all that hard on many filesystems, but it's the interface that's the problem. The ':' character is a perfectly legal filename character, so doing it that way would break things. I think NetApp more or less got the interface right by putting a .snapshot directory in each directory, with time-versioned subdirectories each containing snapshots of that directory's contents at those points in time. It keeps the backups under the same hierarchy as the original files, to avoid permissions headaches, it's accessible over NFS without modifying the client at all, and it's hidden just enough to make it hard for users to do something stupid. If you want to do something like this (and it's generally not a bad idea), make sure you do it in a way that's not going to change the behavior seen by existing applications, and that is accessible to unmodified remote clients. Hidden .snapshot directories are one way, a parallel /backup filesystem could be another, whatever. If you break existing apps, I won't touch it with a ten foot pole. -- Chris -
But to work without breaking userspace it would need to be a character that would pass through any path checking routines, ie be a legal path My personal implementation idea was to store lots of files for the form file:revision_number (I'll keep using that until somebody sugests something better) on the file system itself, with a hard link form the latest version to file (this is probably not a major imporvement and having the hard link coudl make it hard to implement deltas). This could mean no changes to the file system itself (except maybe a flag to say its versioned). The kernel would then do the translation to find the correct file, and would only show the latest version to userapps not The whole interface would be designed to give existing behavior as default for two reasons: users are used to opening a file and getting the latest version and not to break userspace. I personally wouldn't touch this either if it broke userspace. The only userspace change would be the addition of tools to manage the revisions etc. Userspace could later upgrade to take advantage of the new functionality but I cannot see the worth in breaking it. For an example of a working implementation see: http://www.o3one.org/filesystem.html Jack -
I pointed out NetApp's .snapshot directories because that's a method that uses legal path character, but doesn't break anything. With this method, userspace tools will have to be taught that : is suddenly a special character. Userspace already knows that files beginning with . are special and treat them specially. We don't need a new special character for every new feature. We've got one, and it's flexible enough to do what you want, as proven by NetApp's extremely successful implementation. Perhaps you want a slightly different interface from what NetApp has implemented, but what you're suggesting will change the default But what you're talking about *will* break userspace. If I do an ls in a directory, and get pages upon pages of versions of just one file, that's broken. If I tar up a directory and get a tarball that's hundreds of times larger than it should be, that's broken. If you want the files to be hidden from userspace applications that don't know about your backup scheme, (and it sounds like you do) then use the existing convention for hidden files, the prepended '.' This is the universal sign for "don't mess with me unless you know what you're doing". -- Chris -
The idea was that if you did an ls you would get the latest version of the file without the :revision_num. The only visible version would be the latest version, i.e. the current system would not change. The idea was that it would only show earlier versions if they were specifically requested with a :revision_num suffix. In that case the filesystem/kernel would need to recognise the suffix and return the earlier version of the file. The only userspace it would break is files with :num in their name, as I haven't seen any files like that I don't think its too big a problem but the way of specifiying revisions could be changed. Jack -
I have one right now: $ ls /tmp/ksocket-csnook/kdeinit* /tmp/ksocket-csnook/kdeinit__0 /tmp/ksocket-csnook/kdeinit-:0 Note, I did not pass any special arguments to ls to make it pull up that file. You'd have to modify ls to make it do that. You'd also need to modify everything else out there. There are decades of programs out there that would behave differently with the interface you propose. The more fundamental problem with your proposed interface is that it treats a filesystem like an opaque server, instead of a transparent data structure. You want files to be completely invisible to applications that don't know about it, unless the user requests it. Unfortunately, it doesn't work that way. Applications ask for a directory listing, and will open the requested file if and only if the filename in question appears in that listing. If you want to use this opaque server model, you'd be better served putting it in some parallel file system (say, /backup) that won't interfere with naive applications accessing the mundane data. Personally, I like your idea of putting the older versions in the same directory hierarchy, but I think you'd have to use .foo hidden directories to do it right. -- Chris -
The whole idea of the file system is that it wouldn't return the file in the file listing. The user would have to know that the file system was versioning to access the older versions as they would explicitly have to request them. Jack -
Okay, so now you have to modify ls, cp, tar, and thousands of other applications to be aware of the versioning, otherwise you can't use it. Please don't get hung up on the interface. This is a really cool feature that will require some serious engineering work to make it work right. There's no need to reinvent hidden files as well, since we already have a decades-old standard for that. -- Chris -
Chris Snook wrote: I realise that but the way this is designed has quickly become a pain in the past. I think that it would be easier to implement somthing like what was suggested in this email: http://www.ussg.iu.edu/hypermail/linux/kernel/0706.2/1156.html I'm thinking of writing a simple proof of concept version and then seeing what people think from there. Jack -
>>>>> "Jack" == Jack Stone <jack@hawkeye.stone.uk.eu.org> writes: Jack> The whole idea of the file system is that it wouldn't return the Jack> file in the file listing. The user would have to know that the Jack> file system was versioning to access the older versions as they Jack> would explicitly have to request them. So tell me what happens when I do: touch foo:121212121212 -
<joke> This fs crashes and burns because the interface still isn't correct. </joke> Honestly, I don't know and this will be the problem of any interfaces that follow this format. Jack -
You create a new file called 'foo:121212121212'. If you then modify it, you could access the old version as foo:121212121212:0. (The .snapshot directory makes more sense than magic names, since you can see what versions of a file are available without a special tool). -
Jack> The whole idea of the file system is that it wouldn't return the Jack> file in the file listing. The user would have to know that the Jack> file system was versioning to access the older versions as they Matthew> You create a new file called 'foo:121212121212'. If you then Matthew> modify it, you could access the old version as Matthew> foo:121212121212:0. Sure, I knew that, I was trying to push the boundaries a bit here with magic version filenaming conventions to show that it won't scale. Heck, trying to figure out what: touch foo:::::::::::::::::::::: does would be interesting, and would the filesystem parsing code handle that case? Matthew> (The .snapshot directory makes more sense than magic names, Matthew> since you can see what versions of a file are available Matthew> without a special tool). I agree 100%, it's a much better solution, though it has it's own problems, esp over NFS and getting back to your original parent directory properly can be painful. John -
But that would cause havoc with shells which use ; to seperate commands. Using ; would defiantly break userspace Jack -
I think we can all agree on that after this thread but I still don't want people to have the wrong idea about the design. Jack -
I can escape the semicolon just fine in bash. In fact, tab-completion will do this automatically. That's really a non-issue. It just means that anyone who wants to use this feature would have to know what they're doing, which I believe is your goal, right? -- Chris -
I didn't realise this. Would ; break userspace if it was used as the delimiter? This discussion may be academic as this design is looking less and less useful/workable. Jack -
I have no idea. I've never written a file management utility or library, so I Well, I'd argue that the most interesting part of this idea is how it works on the inside. You can implement arbitrarily impractical interfaces to test it out as long as your code is modular enough to implement a community-agreeable interface once it's ready for a wider audience. -- Chris -
Then the user has to quote the ;. BTW `find` use a lone ';' since ages Only for buggy written shell scripts/commands. Who cares? Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services -
On Tue, 19 Jun 2007 12:08:52 -0700 Yes but tdskb:foo.mac[1013,1013,frob];4 is *not* elegant. POSIX is very clear about what is acceptable as magic in a pathname, and the unix spec even more so. The NetApp approach recognizes two important things 1. Old version access is the oddity not the norm 2. Standards behaviour is important Alan -
I think describing VMS pathname syntax as "not elegant" is kind of like 3. An atomic snapshot is more useful than a bunch of disconnected per-file version. Kind of like CVS vs SVN. -hpa -
I believe that since some decision must be made *when* a snapshot is taken, and that should (has to) be done by userspace, a userspace versioning system for doing the backups is the right solution. [1] Whether there is some filesystem (FUSE or native) that allows online browsing of the backups or not is another matter. Ad 1: What userspace needs is - atomic snapshots of complete directory trees, independent of mount boundaries (across filesystems) - an atomic way to change the state of the filesystem for the *whole* system. For FSVS I'll try to use unionfs for that - populate some new directory with my tree of changes, then overmount that over "/", and move the files over one-by-one until the new directory is empty. (Must be checked on reboot, of course). These are actually two similar operations (from the atomic view), but have to be done in completely different ways ... Maybe there could be some "better" interface (if there is one - I don't know what could really be removed from the above workflow). Regards, Phil -
I think I used one of those systems once (or at least another one with .snapshot feature). It managed to completely avoid user space problems by never actually showing .snapshot in directory listings, but you could always cd to it or refer to it explicitly. You never risked having tar or find or anything else accidentally pick it up. Very nice interface. -- Len Sorensen -
since anything starting with . is considered a 'hidden' file per *nix tradition it's ignored by many programs and optionally ignored by most others (and anything that doesn't ignore . files when presending files to the user has many other problems on modern desktop systems anyway ;-) the only trouble I ever had with the .snapshot approach is when tar or find would decend down into the .snapshot when I didn't really intend for it to do so. David Lang -
Netapp optionally made .snapshot not show up in readdir, which solved that problem. I have a bigger issue with it starting with only one dot, which is traditionally used for user configuration information. I think ..snapshot would have been a better choice, and by extension leaving double-dot filenames as filesystem namespace (I know that contradicts POSIX, but the only namespace available in POSIX is /../ which is a global namespace.) -hpa -
Still, anything that depends on increasing the length of file or path names to refer to different versions will encounter problems for long file names and deep paths because there is an upper limit on file and path name lengths. It may be possible to use namespaces where an application can change the view it (and it's children) will have on the storage by switching to a different namespace and tagging it with for instance a specific version or date we're interested in. Not sure if we actually pass namespace information down to the file systems yet though, last time I checked they were only visible for the VFS. Jan -
Then you have the Solaris variant -- rely on openat() flags to decend into another namespace. -hpa -
That is why a readdir shouldn't even include .snapshot. Only explicitly asking to open .snapshot should give it to you. That way tar can't accidentally include it unless you specifically asked it to go into .snapshot and archive the contents. So by hidden it means completely hidden, not just left out by the shell and the unix conventions. Of course backup tools have to know about this, but then again backup tools have to know about all sorts of low level filesystem stuff that normal tools don't. -- Len Sorensen -
I don't know NetApp's implementation, but I assume it is more than just a choice of special character. If you merely start the directory name with a dot, you don't fool anyone but 'ls' and shell wildcard expansion. (And for some enlightened people like me, you don't even fool ls, because we use the --almost-all option to show the dot files by default, having been burned too many times by invisible files). I assume NetApp flags the directory specially so that a POSIX directory read doesn't get it. I've seen that done elsewhere. The same thing, by the way, is possible with Jack's filename:version idea, and I assumed that's what he had in mind. Not that that makes it all OK. -- Bryan Henderson IBM Almaden Research Center San Jose CA Filesystems -
No. The directory is quite visible with a standard 'ls -a'. Instead, they simply mark it as a separate volume/filesystem: i.e. the fsid differs when you call stat(). The whole thing ends up acting rather like our bind mounts. It means that you avoid all those nasty user issues where people try to hard link to/from .snapshot directories, rename files across snapshot boundaries, etc. Trond -
Last I used a Netapp, it was configurable, I believe; I seem to also vaguely remember that one could configure it so that it only was accessible as part of a mount string rather than as part of an already-mounted filesystem. Of course, this was a long time ago. -hpa -
Hmm. So it breaks user space quite a bit. By break, I mean uses that work with more conventional filesystems stop working if you switch to NetAp. Most programs that operate on directory trees willingly cross filesystems, right? Even ones that give you an option, such as GNU cp, don't by default. But if the implementation is, as described, wildly successful, that means users are willing to tolerate this level of breakage, so it could be used for versioning too. But I think I'd rather see a truly hidden directory for this (visible only when looked up explicitly). -- Bryan Henderson IBM Almaden Research Center San Jose CA Filesystems -
Well, if we're going to have super-secret hidden directories, we might as well implement them in a namespace framework. Somebody is going to want generic filesystem namespaces eventually, so having one unified mechanism for doing this kind of thing will make it much easier, especially for userspace apps which would need to be modified to be aware of them. Personally, I'm happy with .snapshot and the like. -- Chris -
<snip> have you looked into ext3cow? it allows you to take snapshots of the entire ext3 fs at a single point, and rollback / extract snapshots at any time later. This may be sufficient for you and the implementation seems to be rather stable already. Cheers, Auke http://www.ext3cow.com/ -
As long as there is only one person using the file system. Rolling back
the entire filesystem may work well for you, but screw up something else
someone else is doing.
And what kind of rights do you have to assign to the user to do that level
of snapshot and rollback? You have to assume that there are more than one
user and that they have less than root privileges.
--
"ANSI C says access to the padding fields of a struct is undefined.
ANSI C also says that struct assignment is a memcpy. Therefore struct
assignment in ANSI C is a violation of ANSI C..."
- Alan Cox
-
Hi, Perhaps BTRFS might be of interest where you can have a subvolume for every user: http://lkml.org/lkml/2007/6/12/242 Hannes -
