Home » Mailing list archives » linux-kernel » 2008 » May » 26

Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.

Previous thread: [PATCH]scsi: check the return value of device_create_file() in powertec.c by Qinghuang Feng on Sunday, May 25, 2008 - 9:18 am. (2 messages)

Next thread: Re: Fwd: [PATCH] kmemcheck: SMP support by Pekka Paalanen on Sunday, May 25, 2008 - 10:30 am. (1 message)
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: <linux-kernel@...>
Cc: <netdev@...>, <linux-fsdevel@...>
Subject: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Sunday, May 25, 2008 - 9:40 am

Hi.


I'm pleased to announce POHMEL high performance network filesystem.

POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System.


Development status can be tracked in filesystem section [1].


This is a high performance network filesystem with local coherent cache of data

and metadata. Its main goal is distributed parallel processing of data.


This release brings following features:

 * Full transaction support for all operations (object creation/removal,

	data reading and writing). Data reading transactions are not optimal yet

	and will be improved in the next release (although fast).

 * Data and metadata cache coherency support. More details on how this is

	implemented one can find in appropriate  section [5].

 * Transaction timeout based resending. If given transaction did not

	receive reply after specified timeout, transaction will be resent

	(possibly to different server).

 * Switched writepage path to ->sendpage() which improved performance and

	robustness of the writing.

 * Preliminary support for parallel data processing. Code to write data

	to multiple servers in parallel and balance reading between them was

	imported, but is not used right now.

 * Fair number of bugfixes.


Basic POHMELFS features:

 * Local coherent (notes [2]) cache for data and metadata.

 * Completely async processing of all events (hard and symlinks are the only

    	exceptions) including object creation and data reading/writing.

 * Flexible object architecture optimized for network processing. Ability to

    	create long pathes to object and remove arbitrary huge directoris in

	single network command.

 * High performance is one of the main design goals.

 * Very fast and scalable multithreaded userspace server. Being in userspace

    	it works with any underlying filesystem and still is much faster than

	async ni-kernel NFS one.

 * Client is able to switch between different servers (if one goes down,

	client automatically reconnects to second and so on).

 * Transactio...
[ message continues ]
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Sunday, May 25, 2008 - 9:41 pm

Often you can capture this information in a more scalable manner, by

having the clients measure an observable _output_ such as latency, over

time.


Each transaction gives the client new feedback about the server(s) being

offline (i.e. timeout), becoming slow, etc.


Another potential tool is having the server(s) send an abstract number,

from 0-100, indicating their level of desire for new { read | write }

transactions.  Let us call this client hint "weight".  Rather than have

a bunch of status messages that indicate server load average, or

do-not-read state, summarize this information into read_weight and

write_weight hints to the client.


With just a few (two?) simple variables, the client is given the ability

to automatically balance and scale load across the cluster, work around

failing servers, etc.


Write balancing need not be overly complex...


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 3:12 am

Server should be able to provide such info because of sudden changes in

environment or administrative tasks. It should not be that many of such

messages, but some info should be broadcasted to connected clients.

Client by itself should also e able to determine if given server is fast


I actually meant only single message with status structure, which can


Yup, some kind of that. I did not yet thought about it in details...


--

	Evgeniy Polyakov

--
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Sunday, May 25, 2008 - 9:15 pm

Is this really the latest commit?


commit 21549d0a101c1c9dcdb1fe8d59685415cc0a5e9a

Author: Evgeniy Polyakov johnpol@2ka.mipt.ru <zbr@gavana.yandex.ru>

Date:   Tue May 13 20:54:49 2008 +0400


I would have guessed modifications occurred more recently than May 13.


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 2:22 am

No it should be cc4d281eb8d9a4d3a5610acf1558c327358f5725

Author: Evgeniy Polyakov johnpol@2ka.mipt.ru <zbr@gavana.yandex.ru>

Date:   Sun May 25 17:24:56 2008 +0400


    Kconfig entries for cc group and debugging.


I used 'git push remote_repo' to push data after repository was create, probably I

should use 'git push --all remote_repo', please try again.


--

	Evgeniy Polyakov

--
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 2:33 am

Nope.


Did you forget git-update-server-info, since your method is HTTP?


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 3:07 am

At least on server HEAD (refs/heads/master) points to

cc4d281eb8d9a4d3a5610acf1558c327358f5725


I tries git-update-server-info on client and pushed all again, but it

returned that everything is up-to-date.


--

	Evgeniy Polyakov

--
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 3:27 am

git-update-server-info needs to be run on the server, not the client.


You can automate this by running this on the server:


	chmod +x .git/hooks/post-update


so that git-update-server-info always runs on the server, after each push.


See git-update-server-info(1) for more info.


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 3:44 am

Hmm... Nothing can be run on server, since git does not believe it is in

git repo:

$ strace git-update-server-info

execve("/usr/bin/git-update-server-info", ["git-update-server-info"], [/* 20 vars */]) = 0


... ommitting mapping and shared lib loading ...


getcwd("/home/s0mbre/public_html/archive/pohmelfs/pohmelfs.git", 4097) = 55

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

access(".git/refs/", X_OK)              = -1 ENOENT (No such file or directory)

chdir("..")                             = 0

write(2, "fatal: ", 7fatal: )                  = 7

write(2, "Not a git repository", 20Not a git repository)    = 20

write(2, "\n", 1


Obviously there is no .git dir, since we are in it:


pohmelfs.git$ ls -l

total 14

-rw-r--r--  1 s0mbre s0mbre   23 May 13 21:33 HEAD

drwxr-xr-x  2 s0mbre s0mbre   48 May 13 21:33 branches

-rw-r--r--  1 s0mbre s0mbre   66 May 13 21:31 config

-rw-r--r--  1 s0mbre s0mbre   58 May 13 21:33 description

drwxr-xr-x  2 s0mbre s0mbre  328 May 13 21:33 hooks

drwxr-xr-x  2 s0mbre s0mbre   96 May 13 21:33 info

drwxr-xr-x 67 s0mbre s0mbre 1608 May 26 10:20 objects

drwxr-xr-x  5 s0mbre s0mbre  120 May 26 10:20 refs


And it can not be accessed from higher layer, since its name is not

'.git', but 'pohmelfs.git'.


pohmelfs.git$ ls -l ../

total 440

-rw-r--r-- 1 s0mbre s0mbre  ...
[ message continues ]
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 3:51 am

Sorry, I should have mentioned, you need to set GIT_DIR environment

variable, for this case.  e.g.


	cd /home/s0mbre/archive/pohmelfs

	GIT_DIR=pohmelfs.git git-update-server-info


or


	cd /home/s0mbre/archive/pohmelfs/pohmelfs.git

	GIT_DIR=. git-update-server-info


Regards,


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Monday, May 26, 2008 - 5:13 am

Excellent, I've updated server info.

Thanks!


--

	Evgeniy Polyakov

--
[view original message]
From: Jeff Garzik <jeff@...>
To: Evgeniy Polyakov <johnpol@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Sunday, May 25, 2008 - 3:55 pm

Is the server available via git?


http://tservice.net.ru/~s0mbre/archive/pohmelfs/server/ does not include

history information, etc.


	Jeff


--
[view original message]
From: Evgeniy Polyakov <johnpol@...>
To: Jeff Garzik <jeff@...>
Cc: <linux-kernel@...>, <netdev@...>, <linux-fsdevel@...>
Subject: Re: POHMELFS high performance network filesystem. Cache coherency, transactions, parallels.
Date: Sunday, May 25, 2008 - 4:21 pm

Hi.


I've just cloned a git tree for the server, but do not expect intersting

stuff there, usually I make big commits which incorporate lots of stuff

in one go.


http://tservice.net.ru/~s0mbre/archive/pohmelfs/server/fserver.git/


--

	Evgeniy Polyakov

--
Previous thread: [PATCH]scsi: check the return value of device_create_file() in powertec.c by Qinghuang Feng on Sunday, May 25, 2008 - 9:18 am. (2 messages)

Next thread: Re: Fwd: [PATCH] kmemcheck: SMP support by Pekka Paalanen on Sunday, May 25, 2008 - 10:30 am. (1 message)