"These patches add local caching for network filesystems such as NFS," began David Howells describing an updated set of thirty-seven patches to introduce FS-Cache. When asked how the patches affect performance, he noted that this was dependent on the use case, highlighting issues when dealing with lots of metadata, "getting metadata from the local disk fs is slower than pulling it across an unshared gigabit ethernet from a server that already has it in memory."
David continued "these points don't mean that fscache is no use, just that you have to consider carefully whether it's of use to *you* given your particular situation, and that depends on various factors," adding, "note that currently FS-Caching is disabled for individual NFS files opened for writing as there's no way to handle the coherency problems thereby introduced." He concluded with a number of simple performance benchmarks.
From: David Howells <dhowells@...>
Subject: [PATCH 00/37] Permit filesystem local caching
Date: Feb 20, 12:05 pm 2008
These patches add local caching for network filesystems such as NFS.
The patches can roughly be broken down into a number of sets:
(*) 01-keys-inc-payload.diff
(*) 02-keys-search-keyring.diff
(*) 03-keys-callout-blob.diff
Three patches to the keyring code made to help the CIFS people.
Included because of patches 05-08.
(*) 04-keys-get-label.diff
A patch to allow the security label of a key to be retrieved.
Included because of patches 05-08.
(*) 05-security-current-fsugid.diff
(*) 06-security-separate-task-bits.diff
(*) 07-security-subjective.diff
(*) 08-security-kernel_service-class.diff
(*) 09-security-kernel-service.diff
(*) 10-security-nfsd.diff
Patches to permit the subjective security of a task to be overridden.
All the security details in task_struct are decanted into a new struct
that task_struct then has two pointers two: one that defines the
objective security of that task (how other tasks may affect it) and one
that defines the subjective security (how it may affect other objects).
Note that I have dropped the idea of struct cred for the moment. With
the amount of stuff that was excluded from it, it wasn't actually any
use to me. However, it can be added later.
Required for cachefiles.
(*) 11-release-page.diff
(*) 12-fscache-page-flags.diff
(*) 13-add_wait_queue_tail.diff
(*) 14-fscache.diff
Patches to provide a local caching facility for network filesystems.
(*) 15-cachefiles-ia64.diff
(*) 16-cachefiles-ext3-f_mapping.diff
(*) 17-cachefiles-write.diff
(*) 18-cachefiles-monitor.diff
(*) 19-cachefiles-export.diff
(*) 20-cachefiles.diff
Patches to provide a local cache in a directory of an already mounted
filesystem.
(*) 21-nfs-comment.diff
(*) 22-nfs-fscache-option.diff
(*) 23-nfs-fscache-kconfig.diff
(*) 24-nfs-fscache-top-index.diff
(*) 25-nfs-fscache-server-obj.diff
(*) 26-nfs-fscache-super-obj.diff
(*) 27-nfs-fscache-inode-obj.diff
(*) 28-nfs-fscache-use-inode.diff
(*) 29-nfs-fscache-invalidate-pages.diff
(*) 30-nfs-fscache-iostats.diff
(*) 31-nfs-fscache-page-management.diff
(*) 32-nfs-fscache-read-context.diff
(*) 33-nfs-fscache-read-fallback.diff
(*) 34-nfs-fscache-read-from-cache.diff
(*) 35-nfs-fscache-store-to-cache.diff
(*) 36-nfs-fscache-mount.diff
(*) 37-nfs-fscache-display.diff
Patches to provide NFS with local caching.
A couple of questions on the NFS iostat changes: (1) Should I update the
iostat version number; (2) is it permitted to have conditional iostats?
I've brought the patchset up to date with respect to the 2.6.25-rc1 merge
window, in particular altering Smack to handle the split in objective and
subjective security in the task_struct.
--
A tarball of the patches is available at:
http://people.redhat.com/~dhowells/fscache/patches/nfs+fscache-30.tar.bz2
To use this version of CacheFiles, the cachefilesd-0.9 is also required. It
is available as an SRPM:
http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9-1.fc7.src.rpm
Or as individual bits:
http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9.tar.bz2
http://people.redhat.com/~dhowells/fscache/cachefilesd.fc
http://people.redhat.com/~dhowells/fscache/cachefilesd.if
http://people.redhat.com/~dhowells/fscache/cachefilesd.te
http://people.redhat.com/~dhowells/fscache/cachefilesd.spec
The .fc, .if and .te files are for manipulating SELinux.
David
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Daniel Phillips <phillips@...>
Subject: Re: [PATCH 00/37] Permit filesystem local caching
Date: Feb 20, 11:07 pm 2008
Hi David,
On Wednesday 20 February 2008 08:05, David Howells wrote:
> These patches add local caching for network filesystems such as NFS.
Have you got before/after benchmark results?
Regards,
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: David Howells <dhowells@...>
Subject: Re: [PATCH 00/37] Permit filesystem local caching
Date: Feb 21, 10:55 am 2008
Daniel Phillips <phillips@phunq.net> wrote:
> Have you got before/after benchmark results?
See attached.
These show a couple of things:
(1) Dealing with lots of metadata slows things down a lot. Note the result of
looking and reading lots of small files with tar (the last result). The
NFS client has to both consult the NFS server *and* the cache. Not only
that, but any asynchronicity the cache may like to do is rendered
ineffective by the fact tar wants to do a read on a file pretty much
directly after opening it.
(2) Getting metadata from the local disk fs is slower than pulling it across
an unshared gigabit ethernet from a server that already has it in memory.
These points don't mean that fscache is no use, just that you have to consider
carefully whether it's of use to *you* given your particular situation, and
that depends on various factors.
Note that currently FS-Caching is disabled for individual NFS files opened for
writing as there's no way to handle the coherency problems thereby introduced.
David
---
===========================
FS-CACHE FOR NFS BENCHMARKS
===========================
(*) The NFS client has a 1.86GHz Core2 Duo CPU and 1GB of RAM.
(*) The NFS client has a Seagate ST380211AS 80GB 7200rpm SATA disk on an
interface running in AHCI mode. The chipset is an Intel G965.
(*) A partition of approx 4.5GB is committed to caching, and is formatted as
Ext3 with a blocksize of 4096 and directory indices.
(*) The NFS client is using SELinux.
(*) The NFS server is running an in-kernel NFSd, and has a 2.66GHz Core2 Duo
CPU and 6GB of RAM. The chipset is an Intel P965.
(*) The NFS client is connected to the NFS server by Gigabit Ethernet.
(*) The NFS mount is made with defaults for all options not relating to the
cache:
warthog:/warthog /warthog nfs
rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,
retrans=2,sec=sys,fsc,addr=w.x.y.z 0 0
==================
FEW BIG FILES TEST
==================
Where:
(*) The NFS server has two files:
[root@andromeda ~]# ls -l /warthog/bigfile
-rw-rw-r-- 1 4043 4043 104857600 2006-11-30 09:39 /warthog/bigfile
[root@andromeda ~]# ls -l /warthog/biggerfile
-rw-rw-r-- 1 4043 4041 209715200 2006-03-21 13:56 /warthog/biggerfile
Both of which are in memory on the server in all cases.
No patches, cold NFS cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m1.909s
user 0m0.000s
sys 0m0.520s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m3.750s
user 0m0.000s
sys 0m0.904s
CONFIG_FSCACHE=n, cold NFS cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.003s
user 0m0.000s
sys 0m0.124s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.100s
user 0m0.004s
sys 0m0.488s
Cold NFS cache, no disk cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.084s
user 0m0.000s
sys 0m0.136s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.020s
user 0m0.000s
sys 0m0.720s
Completely cold caches:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.412s
user 0m0.000s
sys 0m0.892s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.449s
user 0m0.000s
sys 0m2.300s
Warm NFS pagecache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m0.067s
user 0m0.000s
sys 0m0.064s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m0.133s
user 0m0.000s
sys 0m0.136s
Warm Ext3 pagecache, cold NFS pagecache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m0.173s
user 0m0.000s
sys 0m0.172s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m0.316s
user 0m0.000s
sys 0m0.316s
Warm on-disk cache, cold pagecaches:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m1.955s
user 0m0.000s
sys 0m0.244s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m3.596s
user 0m0.000s
sys 0m0.460s
===================================
MANY SMALL/MEDIUM FILE READING TEST
===================================
Where:
(*) The NFS server has an old kernel tree:
[root@andromeda ~]# du -s /warthog/aaa
347340 /warthog/aaa
[root@andromeda ~]# find /warthog/aaa | wc -l
20443
All of which is in memory on the server in all cases.
No patches, cold NFS cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m21.698s
user 0m0.156s
sys 0m5.284s
CONFIG_FSCACHE=n, cold NFS cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m22.337s
user 0m0.152s
sys 0m5.476s
Cold NFS cache, no disk cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m22.734s
user 0m0.124s
sys 0m5.796s
Completely cold caches:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m37.497s
user 0m0.248s
sys 0m6.648s
Warm NFS pagecache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m15.167s
user 0m0.168s
sys 0m4.856s
Warm Ext3 pagecache, cold NFS pagecache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
real 0m13.818s
user 0m0.200s
sys 0m5.492s
Warm on-disk cache, cold pagecaches:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 1m54.350s
user 0m0.044s
sys 0m1.256s
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Wow, Linux just got another
Wow, Linux just got another feature Solaris had for... ages?
Yes, Solaris has had it for 10+ years
However, it generally doesn't improve anything and can often hurt performance. In these days of 1Gbps and even 10Gbps networks and SAN systems, it's mostly a non-starter.
Non Starter
In regards to being a non starter it does have its uses. In a cluster computing environment if you have 1000 nodes traversing the same dataset over 100,000 jobs local caching makes a huge difference.
Wow, Linux just got another
Wow, Linux just got another feature Solaris had for... ages?
Yeah, incredible isn't it?
I agree. This kind of foolish attempt to develop things that might be good for Linux as well must stop. I mean, what's the point of progress? Especially if it's progress already made by someone else, before.
Obviously, all focus should be spent on either doing nothing, or at the very least only do things nobody else has ever done before, regardless of whether it might be useful or not.
Is that what you're saying?
If not, I fail to see what your point might be.
Someone else had this before. So f*cking what? If it's useful, it's useful.
Moron.
(Sorry, I'm a bit cranky today, and your tone got to me. Now please go and troll somewhere else. Stop polluting this place with your presence.)
Local cache slower than remote data
Oh, it sounds kinda dumb that the local cache is slower than accessing remote data over a network.
How is that? Why is that?
Kinda removes the whole purpose of having a cache in the first place.
servers aren't always fast
It's a given today that accessing files over a fast idle network to a powerful server will be faster than accessing them locally on an average hard drive. Especially in this case since it says the server has the files cached in RAM.
The cache becomes really useful when you have a slower network, a busy network or a busy server.