I don't think it will work, only subscribers can post to the DFly groups,

    but we'll muddle through it :-)  I will include the whole of the previous

    posting so the DFly groups see the whole thing, if you continue to get

    bounces.
    I believe I have successfully added you as an 'alias address' to the
    Reading this and a little more that you describe later let me make

    sure I understand the forward-logging methodology you are using.

    You would have multiple individually-tracked transactions in

    progress due to parallelism in operations initiated by userland and each

    would be considered committed when the forward-log logs the completion

    of that particular operation?
    If the forward log entries are not (all) cached in-memory that would mean

    that accesses to the filesystem would have to be run against the log

    first (scanning backwards), and then through to the B-Tree?  You

    would solve the need for having an atomic commit ('flush groups' in

    HAMMER), but it sounds like the algorithmic complexity would be

    very high for accessing the log.
    And even though you wouldn't have to group transactions into larger

    commits the crash recovery code would still have to implement those

    algorithms to resolve directory and file visibility issues.  The problem

    with namespace visibility is that it is possible to create a virtually

    unending chain of separate but inter-dependant transactions which either

    all must go, or none of them.  e.g. creating a, a/b, a/b/c, a/b/x, a/b/c/d,

    etc etc.  At some point you have to be able to commit so the whole mess

    does not get undone by a crash, and many completed mini-transactions

    (file or directory creates) actually cannot be considered complete until

    their governing parent directories (when creating) or children (when

    deleting) have been committed.  The problem can become very complex.
    Last question here:  Your are forward-logging high level operations.

    ...
[ message continues ]

Cache layering is a central idea in the Tux3 atomic update model.  The

cache "front end" consists of data blocks in inode page cache, inode

attributes in inode cache and file names in dentry cache.  The

cache "back end" consists of cached file index blocks, inode table

blocks, and inode table index blocks.  Applications directly modify the

front end cache via syscalls and memory maps, while the back end cache

is modified only by the filesystem during the process of encoding

changes in cache permanently to disk.
The great promise of such a layering is to allow "bumpless" operation.

So long as sufficient cache memory is available and any needed metadata

blocks have been read into cache, the front end does not need to wait

for the back end to complete its work.  It just returns immediately to

its caller after updating a few VFS cache objects, without needing to

locate and update cached disk blocks as well.
In broad outline, the concept is simple, clean and compelling.  In

practice, there are issues to overcome.  First, some background.
Changes made to front end cache are batched into "deltas"[1], where each

delta comprises all the changes required to represent some set of file

operations carried out by the front end, or equivalently, the changes

required to make the filesystem state of the previous delta represent

the cache state as of the new delta.
Each new delta goes through a "setup" step that selects and assigns disk

addresses for updated data blocks, modifies cached index blocks

accordingly, and creates log blocks to specify index block changes

logically.  Following setup, the block images of a delta are

transferred to disk.  Finally a delta commit block is written to

transition the Tux3 volume atomically from one consistent state to the

next.
When the front end hands off a delta to the back end it may not further

modify any blocks of that delta until disk transfer has completed.  The

concept of cache forking was introduced to avoid stalls where...
[ message continues ]

Greetings,
I have zero knowledge as far as filesystems are concerned but I'm

interested in learning what Tux3 is and how it works.

I started digging into the Tux3 mailing list but, truth is, that I

couldn't follow most of the threads.

I also tried reading related articles like:

http://lwn.net/Articles/288896/

http://tux3.org/shapor-tux3/doc/design.html

but I didn't get far either.
Apart from basic and unrelated stuff (like how B-trees work, etc.),

can you propose me some basic reading on filesystem design so that I

can, at least, follow some basic discussion on the mailing list and

start checking out the code?
As a matter of fact, I think that publishing such a list on the

tux3.org would motivate many interested developers on 'joining' the

project.
Thanks :)
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3

--00148536e6dab38a0104788033f5

Content-Type: text/plain; charset=ISO-8859-1
hello sir.....

              myself Nandan, i had completed my engineering in Computer

Science. sir i am beginner to filesystem. i want to understand tux3

filesystem pl help me.

i want to understand the source. from where should i begin to trace the code

pl help me sir.........
--00148536e6dab38a0104788033f5

Content-Type: text/html; charset=ISO-8859-1

Content-Transfer-Encoding: quoted-printable
hello sir.....<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 myself Nandan, i =

had completed my engineering in Computer Science. sir i am beginner to file=

system. i want to understand tux3 filesystem pl help me. <br>i want to unde=

rstand the source. from where should i begin to trace the code pl help me s=

ir.........
--00148536e6dab38a0104788033f5--

I'd like to point out, that if you create a file, open it, then delete

it, you can then still use it to store temporary data - this is indeed

a common use case.  However the amount of data storage may very well

exceed what you would be willing to store in ram, and thus you would

want to be able to write this data out to disk, even though the file

itself doesn't exist any more...  Some sort of swap-like behaviour???
Maciej
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

You mean an orphan temporary file?  I think we just need to make sure

that works as it is supposed to.  It is reasonable for file data of

such a file to be transferred to disk just like any other file, even

though the file is unlinked.  We just need to be sure that it will get

cleaned up like any other orphan.
Daniel
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

I understood that one of the benefits of deferred creation, was that a

later deletion could possibly end up with no disk i/o.  I was just

pointing out, that this is still not quite the case, since we need to

have enough data to later release the files data blocks... although I

guess a deleted files data-blocks could be allocated while only

marking their 'use' in in-memory-state (never writing it to disk).

However, this seems highly error-prone and not worth it.  As such the

above optimization can only really be done if we're deleting a file to

which there are no more open references...
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

The data blocks will only be in the page cache (block cache in

userspace) and not even be assigned physical addresses before the

temporary file is deleted.  So they will just be removed from cache

by the VFS.  It would be pretty useless if only empty files could hit

this optimization.
Regards,
Daniel
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

Ah, but that's precisely my point... having this data only in ram,

without the possibility to write it out to disk defeats the purpose of

using just-deleted files as disk-backed temporary storage.
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

Nothing prevents the file data and inode of an unlinked file from being

written to disk.  The file just has to exist at delta transition time,

linked or not.  If unlinked, the file is an orphan, which has to be

handled in any case.
Regards,
Daniel
_______________________________________________

Tux3 mailing list

Tux3@tux3.org

http://tux3.org/cgi-bin/mailman/listinfo/tux3

linSubscribed now, everything should be OK.
Yes.  Writes tend to be highly parallel in Linux because they are

mainly driven by the VMM attempting to clean cache dirtied by active

writers, who generally do not wait for syncing.  So this will work

really well for buffered IO, which is most of what goes on in Linux.

I have not thought much about how well this works for O_SYNC or

O_DIRECT from a single process.  I might have to do it slightly

differently to avoid performance artifacts there, for example, guess

where the next few direct writes are going to land based on where the

most recent ones did and commit a block that says "the next few commit

blocks will be found here, and here, and here...".
When a forward commit block is actually written it contains a sequence

number and a hash of its transaction in order to know whether the

commit block write ever completed.  This introduces a risk that data

overwritten by the commit block might contain the same hash and same

sequence number in the same position, causing corruption on replay.

The chance of this happening is inversely related to the size of the

hash times the chance of colliding with the same sequence number in

random data times the chance of of rebooting randomly.  So the risk can

be set arbitrarily small by selecting the size of the hash, and using

a good hash.  (Incidentally, TEA was not very good when I tested it in

the course of developing dx_hack_hash for HTree.)
Note: I am well aware that a debate will ensue about whether there is

any such thing as "acceptable risk" in relying on a hash to know if a

commit has completed.  This occurred in the case of Graydon Hoare's

Monotone version control system and continues to this day, but the fact

is, the cool modern version control systems such as Git and Mercurial

now rely very successfully on such hashes.  Nonetheless, the debate

will keep going, possibly as FUD from parties who just plain want to

use some other filesystem for their own reasons.  To quell that

definitively...
[ message continues ]