KernelTrap - NFS

NFS mount failure

Mon, 02 Nov 2009 18:53:54 +0000

I have a problem I'm hoping someone can help me with...

I have an NFS server/client setup. On the server side, I export a romfs filesystem to the NFS. The client computers boot via PXE bootloader and mount their root filesystem via NFS, using tempfs/unionfs to make parts of the filesystem read/write (i.e., /etc, /var, etc)

Parallel Optimized Host Message Exchange Layered File System

Jeremy — Wed, 14 May 2008 16:45:23 +0000

"I'm please to announce [the] POHMEL high performance network filesystem. POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System," began Evgeniy Polyakov, explaining:

"This is a high performance network filesystem with local coherent cache of data and metadata. Its main goal is distributed parallel processing of data. Network filesystem is a client transport. POHMELFS protocol was proven to be superior to NFS in lots (if not all, then it is in a roadmap) operations."

This latest release prompted Jeff Garzik to reply, "this continues to be a neat and interesting project :)" New features include fast transactions, round-robin failover, and near-wire limit performance. This adds to existing features which include a local coherent data and metadata cache, async processing of most events, and a fast and scalable multi threaded user space server. Planned features include a server extension to allow mirroring data across multiple devices, strong authentication, and possible data encryption when transferring data over the network. Evgeniy linked to several benchmarks in his blog.

Local Caching For Network Filesystems

Jeremy — Thu, 21 Feb 2008 15:35:41 +0000

"These patches add local caching for network filesystems such as NFS," began David Howells describing an updated set of thirty-seven patches to introduce FS-Cache. When asked how the patches affect performance, he noted that this was dependent on the use case, highlighting issues when dealing with lots of metadata, "getting metadata from the local disk fs is slower than pulling it across an unshared gigabit ethernet from a server that already has it in memory."

David continued "these points don't mean that fscache is no use, just that you have to consider carefully whether it's of use to *you* given your particular situation, and that depends on various factors," adding, "note that currently FS-Caching is disabled for individual NFS files opened for writing as there's no way to handle the coherency problems thereby introduced." He concluded with a number of simple performance benchmarks.

Swap Over NFS

Jeremy — Sat, 03 Nov 2007 03:07:19 +0000

"The problem with swap over network is the generic swap problem: needing memory to free memory. Normally this is solved using mempools, as can be seen in the BIO layer," explained Peter Zijlstra. "Swap over network has the problem that the network subsystem does not use fixed sized allocations, but heavily relies on kmalloc(). This makes mempools unusable."

The first fifteen patches set up a generic framework for reserving memory. Patches 16-23 actually put the framework to use on the network stack. Peter noted, "a network write back completion [involves] receiving packets, which when there is no memory, is rather hard. And even when there is memory there is no guarantee that the required packet comes in in the window that that memory buys us." He went on to explain, "the solution to this problem is found in the fact that network is to be assumed lossy. Even now, when there is no memory to receive packets the network card will have to discard packets. What we do is move this into the network stack." Patches 24-26 set up an infrastructure for swapping to a filesystem instead of a block device, which is then utilized by the final patches, "finally, convert NFS to make use of the new network and vm infrastructure to provide swap over NFS." When the usefulness of these patches were questioned, Peter noted, "There is a large corporate demand for this, which is why I'm doing this. The typical usage scenarios are: 1) cluster/blades, where having local disks is a cost issue (maintenance of failures, heat, etc) 2) virtualisation, where dumping the storage on a networked storage unit makes for trivial migration and what not.."

Memory Management Improvements

Jeremy — Sat, 27 Oct 2007 10:04:59 +0000

A recent report on the lkml suggested improved IO/writeback performance in the recently released 2.6.24-rc1 kernel compared to the earlier 2.6.19.2 and 2.6.22.6 kernels. Credit was given to some patches by Peter Zijlstra. Ingo Molnar replied, "wow, really nice results! Peter does know how to make stuff fast :) Now lets pick up some of Peter's other, previously discarded patches as well :-)" He pointed to several patches "as a starter", then quipped, "I think the MM should get out of deep-feature-freeze mode - there's tons of room to improve :-/"

Andrew Morton replied, "kidding. We merged about 265 MM patches in 2.6.24-rc1: 482 files changed, 8071 insertions(+), 5142 deletions(-)". He added, "a lot of that was new functionality. That's easier to add than things which change long-standing functionality." Of the patches Ingo pointed to, Peter noted he was currently working on polishing the swap-over-NFS patch, "will post that one again, soonish.... Esp. after Linus professed liking to have swap over NFS." Rik van Riel also replied regarding rewriting the page replacement code, "at the moment I only have the basic 'plumbing' of the split VM working and am fixing some bugs in that. Expect a patch series with that soon, so you guys can review that code and tell me where to beat it into shape some more :)"

NFS Client Updates for 2.6.24

Jeremy — Thu, 04 Oct 2007 12:31:08 +0000

Trond Myklebust noted the NFS client updates for the upcoming 2.6.24 kernel:

"Aside from the usual updates from Chuck for NFS-over-IPv6 (still incomplete) and a number of bugfixes for the text-based mount code, the main news in the NFS tree is the merging of support for the NFS/RDMA client code from Tom Talpey and the NetApp New England (NANE) team."

He continued, "we also have the 64-bit inode support from RedHat/Peter Staubach. There is also the addition of a nfs_vm_page_mkwrite() method in order to clean up the mmap() write code. Finally, I've been working on a number of updates for the attribute revalidation, having pulled apart most of the dentry and attribute revalidation into separate variables. A number of fixes that address existing bugs fell out of that review, which should hopefully result in more efficient dcache behaviour..." Actual source changes can be browsed in the NFS client git repository.

Copy on Write Credentials

Jeremy — Tue, 18 Sep 2007 19:16:19 +0000

"Here's a new version of my credentials patch. It's still very basic, with only Ext3, (V)FAT, NFS, AFS, SELinux and keyrings compiled in on an x86_64 arch kernel," stated David Howells. He described the patch as, "introduce a copy on write credentials record (struct cred). The fsuid, fsgid, supplementary groups list move into it (DAC security). The session, process and thread keyrings are reflected in it, but don't primarily reside there as they aren't per-thread and occasionally need to be instantiated or replaced by other threads or processes."

Casey Schaufler asked, "what I don't really understand is what value is gained by this exercise. Are the savings sufficiently significant to justify the effort?" Trond Myklebust explained, "it is not about savings, but about new functionality. Basically, the existence of reference-counted credentials will allow AFS and NFS to cache that information and use it for deferred writes etc." David added, "and also make it easier for cachefiles and hopefully NFSd to override the active security. There's a comment somewhere in, I think, the SunRPC code in the Linux kernel bemoaning the lack of this very feature:-)"