Local Caching For Network Filesystems

"These patches add local caching for network filesystems such as NFS," began David Howells [1] describing an updated set of thirty-seven patches to introduce FS-Cache [2]. When asked how the patches affect performance, he noted that this was dependent on the use case, highlighting issues when dealing with lots of metadata, "getting metadata from the local disk fs is slower than pulling it across an unshared gigabit ethernet from a server that already has it in memory."
David continued "these points don't mean that fscache is no use, just that you have to consider carefully whether it's of use to *you* given your particular situation, and that depends on various factors," adding, "note that currently FS-Caching is disabled for individual NFS files opened for writing as there's no way to handle the coherency problems thereby introduced." He concluded with a number of simple performance benchmarks.
From: David Howells <dhowells@...>
Subject: [PATCH 00/37] Permit filesystem local caching
 [2]Date: Feb 20, 12:05 pm 2008

These patches add local caching for network filesystems such as NFS.

The patches can roughly be broken down into a number of sets:

  (*) 01-keys-inc-payload.diff
  (*) 02-keys-search-keyring.diff
  (*) 03-keys-callout-blob.diff

      Three patches to the keyring code made to help the CIFS people.
      Included because of patches 05-08.

  (*) 04-keys-get-label.diff

      A patch to allow the security label of a key to be retrieved.
      Included because of patches 05-08.

  (*) 05-security-current-fsugid.diff
  (*) 06-security-separate-task-bits.diff
  (*) 07-security-subjective.diff
  (*) 08-security-kernel_service-class.diff
  (*) 09-security-kernel-service.diff
  (*) 10-security-nfsd.diff

      Patches to permit the subjective security of a task to be overridden.
      All the security details in task_struct are decanted into a new struct
      that task_struct then has two pointers two: one that defines the
      objective security of that task (how other tasks may affect it) and one
      that defines the subjective security (how it may affect other objects).

      Note that I have dropped the idea of struct cred for the moment.  With
      the amount of stuff that was excluded from it, it wasn't actually any
      use to me.  However, it can be added later.

      Required for cachefiles.

  (*) 11-release-page.diff
  (*) 12-fscache-page-flags.diff
  (*) 13-add_wait_queue_tail.diff
  (*) 14-fscache.diff

      Patches to provide a local caching facility for network filesystems.

  (*) 15-cachefiles-ia64.diff
  (*) 16-cachefiles-ext3-f_mapping.diff
  (*) 17-cachefiles-write.diff
  (*) 18-cachefiles-monitor.diff
  (*) 19-cachefiles-export.diff
  (*) 20-cachefiles.diff

      Patches to provide a local cache in a directory of an already mounted
      filesystem.

  (*) 21-nfs-comment.diff
  (*) 22-nfs-fscache-option.diff
  (*) 23-nfs-fscache-kconfig.diff
  (*) 24-nfs-fscache-top-index.diff
  (*) 25-nfs-fscache-server-obj.diff
  (*) 26-nfs-fscache-super-obj.diff
  (*) 27-nfs-fscache-inode-obj.diff
  (*) 28-nfs-fscache-use-inode.diff
  (*) 29-nfs-fscache-invalidate-pages.diff
  (*) 30-nfs-fscache-iostats.diff
  (*) 31-nfs-fscache-page-management.diff
  (*) 32-nfs-fscache-read-context.diff
  (*) 33-nfs-fscache-read-fallback.diff
  (*) 34-nfs-fscache-read-from-cache.diff
  (*) 35-nfs-fscache-store-to-cache.diff
  (*) 36-nfs-fscache-mount.diff
  (*) 37-nfs-fscache-display.diff

      Patches to provide NFS with local caching.

      A couple of questions on the NFS iostat changes: (1) Should I update the
      iostat version number; (2) is it permitted to have conditional iostats?


I've brought the patchset up to date with respect to the 2.6.25-rc1 merge
window, in particular altering Smack to handle the split in objective and
subjective security in the task_struct.

--
A tarball of the patches is available at:

	http://people.redhat.com/~dhowells/fscache/patches/nfs+fscache-30.tar.bz2 [3]


To use this version of CacheFiles, the cachefilesd-0.9 is also required.  It
is available as an SRPM:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9-1.fc7.src.rpm [4]

Or as individual bits:

	http://people.redhat.com/~dhowells/fscache/cachefilesd-0.9.tar.bz2 [5]
	http://people.redhat.com/~dhowells/fscache/cachefilesd.fc [6]
	http://people.redhat.com/~dhowells/fscache/cachefilesd.if [7]
	http://people.redhat.com/~dhowells/fscache/cachefilesd.te [8]
	http://people.redhat.com/~dhowells/fscache/cachefilesd.spec [9]

The .fc, .if and .te files are for manipulating SELinux.

David
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [10]
More majordomo info at  http://vger.kernel.org/majordomo-info.html [11]


From: Daniel Phillips <phillips@...>
Subject: Re: [PATCH 00/37] Permit filesystem local caching
 [11]Date: Feb 20, 11:07 pm 2008

Hi David,

On Wednesday 20 February 2008 08:05, David Howells wrote:
> These patches add local caching for network filesystems such as NFS.

Have you got before/after benchmark results?

Regards,

Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [12]
More majordomo info at  http://vger.kernel.org/majordomo-info.html [13]


From: David Howells <dhowells@...>
Subject: Re: [PATCH 00/37] Permit filesystem local caching
 [13]Date: Feb 21, 10:55 am 2008

Daniel Phillips <phillips@phunq.net> wrote:

> Have you got before/after benchmark results?

See attached.

These show a couple of things:

 (1) Dealing with lots of metadata slows things down a lot.  Note the result of
     looking and reading lots of small files with tar (the last result).  The
     NFS client has to both consult the NFS server *and* the cache.  Not only
     that, but any asynchronicity the cache may like to do is rendered
     ineffective by the fact tar wants to do a read on a file pretty much
     directly after opening it.

 (2) Getting metadata from the local disk fs is slower than pulling it across
     an unshared gigabit ethernet from a server that already has it in memory.

These points don't mean that fscache is no use, just that you have to consider
carefully whether it's of use to *you* given your particular situation, and
that depends on various factors.

Note that currently FS-Caching is disabled for individual NFS files opened for
writing as there's no way to handle the coherency problems thereby introduced.

David
---

			  ===========================
			  FS-CACHE FOR NFS BENCHMARKS
			  ===========================

 (*) The NFS client has a 1.86GHz Core2 Duo CPU and 1GB of RAM.

 (*) The NFS client has a Seagate ST380211AS 80GB 7200rpm SATA disk on an
     interface running in AHCI mode.  The chipset is an Intel G965.

 (*) A partition of approx 4.5GB is committed to caching, and is formatted as
     Ext3 with a blocksize of 4096 and directory indices.

 (*) The NFS client is using SELinux.

 (*) The NFS server is running an in-kernel NFSd, and has a 2.66GHz Core2 Duo
     CPU and 6GB of RAM.  The chipset is an Intel P965.

 (*) The NFS client is connected to the NFS server by Gigabit Ethernet.

 (*) The NFS mount is made with defaults for all options not relating to the
     cache:

	warthog:/warthog /warthog nfs
		rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,
		retrans=2,sec=sys,fsc,addr=w.x.y.z 0 0


==================
FEW BIG FILES TEST
==================

Where:

 (*) The NFS server has two files:

	[root@andromeda ~]# ls -l /warthog/bigfile
	-rw-rw-r-- 1 4043 4043 104857600 2006-11-30 09:39 /warthog/bigfile
	[root@andromeda ~]# ls -l /warthog/biggerfile 
	-rw-rw-r-- 1 4043 4041 209715200 2006-03-21 13:56 /warthog/biggerfile

     Both of which are in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.909s
	user    0m0.000s
	sys     0m0.520s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.750s
	user    0m0.000s
	sys     0m0.904s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.003s
	user    0m0.000s
	sys     0m0.124s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.100s
	user    0m0.004s
	sys     0m0.488s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.084s
	user    0m0.000s
	sys     0m0.136s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.020s
	user    0m0.000s
	sys     0m0.720s

Completely cold caches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m2.412s
	user    0m0.000s
	sys     0m0.892s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m4.449s
	user    0m0.000s
	sys     0m2.300s

Warm NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.067s
	user    0m0.000s
	sys     0m0.064s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.133s
	user    0m0.000s
	sys     0m0.136s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m0.173s
	user    0m0.000s
	sys     0m0.172s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m0.316s
	user    0m0.000s
	sys     0m0.316s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
	real    0m1.955s
	user    0m0.000s
	sys     0m0.244s
	[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
	real    0m3.596s
	user    0m0.000s
	sys     0m0.460s


===================================
MANY SMALL/MEDIUM FILE READING TEST
===================================

Where:

 (*) The NFS server has an old kernel tree:

	[root@andromeda ~]# du -s /warthog/aaa
	347340  /warthog/aaa
	[root@andromeda ~]# find /warthog/aaa | wc -l
	20443

     All of which is in memory on the server in all cases.


No patches, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m21.698s
	user    0m0.156s
	sys     0m5.284s

CONFIG_FSCACHE=n, cold NFS cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.337s
	user    0m0.152s
	sys     0m5.476s

Cold NFS cache, no disk cache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m22.734s
	user    0m0.124s
	sys     0m5.796s

Completely cold caches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m37.497s
	user    0m0.248s
	sys     0m6.648s

Warm NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    0m15.167s
	user    0m0.168s
	sys     0m4.856s

Warm Ext3 pagecache, cold NFS pagecache:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	tar: Removing leading `/' from member names
	tar: Removing leading `/' from hard link targets

	real    0m13.818s
	user    0m0.200s
	sys     0m5.492s

Warm on-disk cache, cold pagecaches:

	[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
	real    1m54.350s
	user    0m0.044s
	sys     0m1.256s
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org [14]
More majordomo info at  http://vger.kernel.org/majordomo-info.html [15]
Local Caching For Network Filesystems

Related links:
