Another interesting facts:

According to exports(5) small integers or UUIDs must be used for "fsid=" option.

If I set "fsid=__UUID__" in /etc/exports (where __UUID__ is UUID of partition returned by blkid command), then I got _exactly_ the same error as the first time: impossible to mount nfs partition from Linux client box, and "Stale NFS file handle" while trying to cd into mounted dir on OpenBSD box.

If I set "fsid=1" in /etc/exports, then from Linux client box I can write files without any performance issues, and from OpenBSD client box I get this: I copy a file (size's around 50-60 mibs) after visible full existance on the other side, it freezes and I see nfsrcvl call in top; few mins later I notice nfs_fsy call in top; a few mins later cp returns 0, and file is copied successfully.

I checked sha1sum hashes on both sides, they're equal.

On the server with tcpdump (while writing file from OpenBSD box) it's visible:

22:49:17.002445 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 164)
    172.17.2.2.2049 > 81.200.8.213.1393674899: reply ok 136 write POST: REG 100644 ids 8000/10 sz 27583794 8192 bytes
22:49:17.003105 IP (tos 0x0, ttl 64, id 64645, offset 0, flags [+], proto UDP (17), length 1500)
    81.200.8.213.1015648788 > 172.17.2.2.2049: 1472 write fh Unknown/01000101010000001D080000000000000000000000A4C0000000200000000002 8192 (8192) bytes @ 10797056 <filesync>
22:49:17.003131 IP (tos 0x0, ttl 64, id 64645, offset 1480, flags [+], proto UDP (17), length 1500)
    81.200.8.213 > 172.17.2.2: udp
22:49:17.003345 IP (tos 0x0, ttl 64, id 64645, offset 2960, flags [+], proto UDP (17), length 1500)
    81.200.8.213 > 172.17.2.2: udp
22:49:17.003468 IP (tos 0x0, ttl 64, id 64645, offset 4440, flags [+], proto UDP (17), length 1500)
    81.200.8.213 > 172.17.2.2: udp
22:49:17.003590 IP (tos 0x0, ttl 64, id 64645, offset 5920, flags [+], proto UDP (17), length 1500)
    81.200.8.213 > 172.17.2.2: udp
22:49:17.003598 IP (tos 0x0, ttl 64, id 64645, offset 7400, flags [none], proto UDP (17), length 940)
    81.200.8.213 > 172.17.2.2: udp

No errors, like in the first log. But something's definetely incorrect here.

Also tried mounting the partition with "-T" (tcp) flag on the client side -- no luck.

On Wed, 21 Apr 2010 22:26:12 +0400
Vlad Glagolev <stealth@sourcemage.org> wrote:

quoted text
> Hmm, more testing.. It works only with tiny files flawlessly on OpenBSD (client).
> 
> If a filesize is around 50 mibs, then it just freezes and eats cpu with nfsrcvl call.
> 
> On Linux I don't see such problem. Even big files are transfered with good enough speed.
> 
> On Wed, 21 Apr 2010 21:32:01 +0400
> Vlad Glagolev <stealth@sourcemage.org> wrote:
> 
> > On Wed, 21 Apr 2010 12:09:20 -0500
> > Roger Heflin <rogerheflin@gmail.com> wrote:
> > 
> > > On Wed, Apr 21, 2010 at 11:48 AM, Vlad Glagolev <stealth@sourcemage.org> wrote:
> > > > Thanks for reply, Steve!
> > > >
> > > > parameters are pretty trivial, (rw,insecure) for exports, and defaults while mounting via ``mount host:/path /path'' command.
> > > >
> > > > Yes. That sounds interesting, since XFS works fine with there partitions.
> > > > Also, I must say it's WD20EARS drives (with 4kb sector size, though parted says it's 512b).
> > > >
> > > > I also tried another NFS daemon implementation (cvs version, not .22) -- unfsd (unfs3).
> > > > It mounts ok, but when I try to write any file to the server -- I get the same error (Stale NFS file handle).
> > > >
> > > > And on the server side in dmesg I see this:
> > > >
> > > > --
> > > > NFS: server 172.17.2.2 error: fileid changed
> > > > fsid 0:f: expected fileid 0x2033, got 0xb6d1e05fa150ce09
> > > > NFS: server 172.17.2.2 error: fileid changed
> > > > fsid 0:f: expected fileid 0x2033, got 0x26550b0132c0b1
> > > > NFS: server 172.17.2.2 error: fileid changed
> > > > fsid 0:f: expected fileid 0x2033, got 0x8202a60053000020
> > > > NFS: server 172.17.2.2 error: fileid changed
> > > > fsid 0:f: expected fileid 0x2033, got 0xe542f93ebc8fe157
> > > > NFS: server 172.17.2.2 error: fileid changed
> > > > fsid 0:f: expected fileid 0x2033, got 0xc00cd74ea904301
> > > > --
> > > >
> > > > looks like NFS protocol doesn't like something in partitioned software RAID.
> > > >
> > > 
> > > 
> > > Try manually setting the fsid=something in the exports file and
> > > reexport and remount on the target system, if there was a fsid
> > > collision of some sort then nfs would be hitting the wrong fs...
> > > 
> > > NFS generates the fsid automatically based on the devices major minor,
> > > and it is possible there is something odd about the major minor
> > > numbers that make them not unique...and collide with someone else
> > > major minor.
> > 
> > BAH! How simple!
> > 
> > Thank you very much, Roger!
> > 
> > I've just added fsid=1 (yes, only these few chars) to exports, and it worked! Unbelievable, really :)
> > Of course I've checked it on OpenBSD and Linux under both nfsd and unfsd. Works flawlessly.
> > 
> > Thanks a lot again.
> > 
> > But it seems to be a bug, right? If so, patches welcome.. I'll test it with great pleasure.
> > 
> > -- 
> > Dont wait to die to find paradise...
> > --
> > Cheerz,
> > Vlad "Stealth" Glagolev
> 
> 
> -- 
> Dont wait to die to find paradise...
> --
> Cheerz,
> Vlad "Stealth" Glagolev


-- 
Dont wait to die to find paradise...
--
Cheerz,
Vlad "Stealth" Glagolev