See attached.
These show a couple of things:
(1) Dealing with lots of metadata slows things down a lot. Note the result of
looking and reading lots of small files with tar (the last result). The
NFS client has to both consult the NFS server *and* the cache. Not only
that, but any asynchronicity the cache may like to do is rendered
ineffective by the fact tar wants to do a read on a file pretty much
directly after opening it.
(2) Getting metadata from the local disk fs is slower than pulling it across
an unshared gigabit ethernet from a server that already has it in memory.
These points don't mean that fscache is no use, just that you have to consider
carefully whether it's of use to *you* given your particular situation, and
that depends on various factors.
Note that currently FS-Caching is disabled for individual NFS files opened for
writing as there's no way to handle the coherency problems thereby introduced.
David
---
===========================
FS-CACHE FOR NFS BENCHMARKS
===========================
(*) The NFS client has a 1.86GHz Core2 Duo CPU and 1GB of RAM.
(*) The NFS client has a Seagate ST380211AS 80GB 7200rpm SATA disk on an
interface running in AHCI mode. The chipset is an Intel G965.
(*) A partition of approx 4.5GB is committed to caching, and is formatted as
Ext3 with a blocksize of 4096 and directory indices.
(*) The NFS client is using SELinux.
(*) The NFS server is running an in-kernel NFSd, and has a 2.66GHz Core2 Duo
CPU and 6GB of RAM. The chipset is an Intel P965.
(*) The NFS client is connected to the NFS server by Gigabit Ethernet.
(*) The NFS mount is made with defaults for all options not relating to the
cache:
warthog:/warthog /warthog nfs
rw,vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp,timeo=600,
retrans=2,sec=sys,fsc,addr=w.x.y.z 0 0
==================
FEW BIG FILES TEST
==================
Where:
(*) The NFS server has two files:
[root@andromeda ~]# ls -l /warthog/bigfile
-rw-rw-r-- 1 4043 4043 104857600 2006-11-30 09:39 /warthog/bigfile
[root@andromeda ~]# ls -l /warthog/biggerfile
-rw-rw-r-- 1 4043 4041 209715200 2006-03-21 13:56 /warthog/biggerfile
Both of which are in memory on the server in all cases.
No patches, cold NFS cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m1.909s
user 0m0.000s
sys 0m0.520s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m3.750s
user 0m0.000s
sys 0m0.904s
CONFIG_FSCACHE=n, cold NFS cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.003s
user 0m0.000s
sys 0m0.124s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.100s
user 0m0.004s
sys 0m0.488s
Cold NFS cache, no disk cache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.084s
user 0m0.000s
sys 0m0.136s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.020s
user 0m0.000s
sys 0m0.720s
Completely cold caches:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m2.412s
user 0m0.000s
sys 0m0.892s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m4.449s
user 0m0.000s
sys 0m2.300s
Warm NFS pagecache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m0.067s
user 0m0.000s
sys 0m0.064s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m0.133s
user 0m0.000s
sys 0m0.136s
Warm Ext3 pagecache, cold NFS pagecache:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m0.173s
user 0m0.000s
sys 0m0.172s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m0.316s
user 0m0.000s
sys 0m0.316s
Warm on-disk cache, cold pagecaches:
[root@andromeda ~]# time cat /warthog/bigfile >/dev/null
real 0m1.955s
user 0m0.000s
sys 0m0.244s
[root@andromeda ~]# time cat /warthog/biggerfile >/dev/null
real 0m3.596s
user 0m0.000s
sys 0m0.460s
===================================
MANY SMALL/MEDIUM FILE READING TEST
===================================
Where:
(*) The NFS server has an old kernel tree:
[root@andromeda ~]# du -s /warthog/aaa
347340 /warthog/aaa
[root@andromeda ~]# find /warthog/aaa | wc -l
20443
All of which is in memory on the server in all cases.
No patches, cold NFS cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m21.698s
user 0m0.156s
sys 0m5.284s
CONFIG_FSCACHE=n, cold NFS cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m22.337s
user 0m0.152s
sys 0m5.476s
Cold NFS cache, no disk cache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m22.734s
user 0m0.124s
sys 0m5.796s
Completely cold caches:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m37.497s
user 0m0.248s
sys 0m6.648s
Warm NFS pagecache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 0m15.167s
user 0m0.168s
sys 0m4.856s
Warm Ext3 pagecache, cold NFS pagecache:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
real 0m13.818s
user 0m0.200s
sys 0m5.492s
Warm on-disk cache, cold pagecaches:
[root@andromeda ~]# time tar cf - /warthog/aaa >/dev/zero
real 1m54.350s
user 0m0.044s
sys 0m1.256s
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
