From: Ivan Voras <ivoras@...>
Subject: When will ZFS become stable?
Date: Jan 4, 7:42 am 2008

Hi,

As far as I know about the details of implementation and what would it
take to fix the problems, is it safe to assume ZFS will never become
stable during 7.x lifetime?


From: Brooks Davis <brooks@...>
Subject: Re: When will ZFS become stable?
Date: Jan 4, 12:33 pm 2008

On Fri, Jan 04, 2008 at 12:42:28PM +0100, Ivan Voras wrote:
> Hi,
>=20
> As far as I know about the details of implementation and what would it
> take to fix the problems, is it safe to assume ZFS will never become
> stable during 7.x lifetime?

I suppose that depends what you mean by stable.  It seems stable enough
for a number of applications today.  It's clearly not widely tested
since we haven't shipped a release based on it.  It's possible some of
the issues of memory requirements won't be fixable in 7.x, but I don't
think that's a given.

-- Brooks


From: Ivan Voras <ivoras@...>
Subject: Re: When will ZFS become stable?
Date: Jan 4, 1:58 pm 2008

On 04/01/2008, Brooks Davis <brooks@freebsd.org> wrote:
> On Fri, Jan 04, 2008 at 12:42:28PM +0100, Ivan Voras wrote:
> > Hi,
> >
> > As far as I know about the details of implementation and what would it
> > take to fix the problems, is it safe to assume ZFS will never become
> > stable during 7.x lifetime?
>
> I suppose that depends what you mean by stable.

My yardstick is currently "when a month goes by without anyone
complaining it crashed on him" :)

>It seems stable enough
> for a number of applications today.

This number is not so large. It seems to be easily crashed by rsync,
for example (speaking from my own experience, and also some of my
colleagues).

> It's possible some of
> the issues of memory requirements won't be fixable in 7.x, but I don't
> think that's a given.

I listened to some of Pawel's talks and devsummit brainstormings and I
get the feeling *none* of the problems can be fixed in 7.x, especially
on i386. I'm just asking for more official confirmation.

This is not a trivial question, since it involves deploying systems to
be maintained some years into the future - if ZFS will become stable
relatively shortly, it might be worth putting up with crashes, but if
not, there will be no near-future deployments of it.
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

From: Scott Long <scottl@...>
Subject: ZFS honesty
Date: Jan 6, 5:00 pm 2008

Kris Kennaway wrote:
> Ivan Voras wrote:
>> Kris Kennaway wrote:
>>> Ivan Voras wrote:
>>>> Robert Watson wrote:
>>>>
>>>>> I'm not sure if anyone has mentioned this yet in the thread, but 
>>>>> another thing worth taking into account in considering the 
>>>>> stability of ZFS is whether or not Sun considers it a production 
>>>>> feature in Solaris.  Last I heard, it was still considered an 
>>>>> experimental feature there as well.
>>>>
>>>> Last I heard, rsync didn't crash Solaris on ZFS :)
>>>
>>> [Citation needed]
>>
>> I can't provide citation about a thing that doesn't happen - you don't 
>> hear things like "oh and yesterday I ran rsync on my Solaris with ZFS 
>> and *it didn't crash*!" often.
>>
>> But, with some grains of salt taken, consider this Google results:
>>
>> * searching for "rsync crash solaris zfs": 790 results, most of them 
>> obviously irrelevant
>> * searching for "rsync crash freebsd zfs": 10,800 results; a small 
>> number of the results is from this thread, some are duplicates, but 
>> it's a large number in any case.
>>
>> I feel that the number of Solaris+ZFS installations worldwide is 
>> larger than that of FreeBSD+ZFS and they've had ZFS longer.
> 
> Almost all Solaris systems are 64 bit.
> 
> Kris

So, let's be honest here.  ZFS is simply unreliable on FreeBSD/i386.
There are things that you can do mitigate the problems, and in certain
well controlled environments you might be able to make it work well
enough for your needs.  But as a general rule, don't expect it to work
reliably, period.  This is backed up by Sun's own recommendation to not
run it on 32-bit Solaris.

But let's also be honest about ZFS in the 64-bit world.  There is ample
evidence that ZFS basically wants to grow unbounded in proportion to the
workload that you give it.  Indeed, even Sun recommends basically
throwing more RAM at most problems.  Again, tuning is often needed, and
I think it's fair to say that it can't be expected to work on arbitrary
workloads out of the box.

Now, what about the other problems that have been reported in this
thread by Ivan and others?  I don't think that it can be said that the
only problem that ZFS has is with memory.  Unfortunately, it looks like
these "other" problems aren't well quantified, so I think that they are
being unfairly dismissed.  But at the same time, maybe these other
problems are rare and unique enough that they represent very special
cases that won't be encountered by most people.  But it also tells me
that ZFS is still immature, at least in FreeBSD.

The universal need for tuning combined with the poorly understood
problem reports tells me that administrators considering ZFS should
expect to spend a fair amount of timing testing and tuning.  Don't
expect it to work out of the box for your situation.  That's not to
say that it's useless; there are certainly many people who can attest to
it working well for them.  Just be prepared to spend time and possibly
money making it work, and be willing to provide good problem reports for
any non-memory related problems that you encounter.

Scott
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"


From: Kris Kennaway <kris@...>
Subject: Re: ZFS honesty
Date: Jan 6, 5:32 pm 2008

Scott Long wrote:
> Kris Kennaway wrote:
>> Ivan Voras wrote:
>>> Kris Kennaway wrote:
>>>> Ivan Voras wrote:
>>>>> Robert Watson wrote:
>>>>>
>>>>>> I'm not sure if anyone has mentioned this yet in the thread, but 
>>>>>> another thing worth taking into account in considering the 
>>>>>> stability of ZFS is whether or not Sun considers it a production 
>>>>>> feature in Solaris.  Last I heard, it was still considered an 
>>>>>> experimental feature there as well.
>>>>>
>>>>> Last I heard, rsync didn't crash Solaris on ZFS :)
>>>>
>>>> [Citation needed]
>>>
>>> I can't provide citation about a thing that doesn't happen - you 
>>> don't hear things like "oh and yesterday I ran rsync on my Solaris 
>>> with ZFS and *it didn't crash*!" often.
>>>
>>> But, with some grains of salt taken, consider this Google results:
>>>
>>> * searching for "rsync crash solaris zfs": 790 results, most of them 
>>> obviously irrelevant
>>> * searching for "rsync crash freebsd zfs": 10,800 results; a small 
>>> number of the results is from this thread, some are duplicates, but 
>>> it's a large number in any case.
>>>
>>> I feel that the number of Solaris+ZFS installations worldwide is 
>>> larger than that of FreeBSD+ZFS and they've had ZFS longer.
>>
>> Almost all Solaris systems are 64 bit.
>>
>> Kris
> 
> So, let's be honest here.  ZFS is simply unreliable on FreeBSD/i386.
> There are things that you can do mitigate the problems, and in certain
> well controlled environments you might be able to make it work well
> enough for your needs.  But as a general rule, don't expect it to work
> reliably, period.  This is backed up by Sun's own recommendation to not
> run it on 32-bit Solaris.
> 
> But let's also be honest about ZFS in the 64-bit world.  There is ample
> evidence that ZFS basically wants to grow unbounded in proportion to the
> workload that you give it.  Indeed, even Sun recommends basically
> throwing more RAM at most problems.  Again, tuning is often needed, and
> I think it's fair to say that it can't be expected to work on arbitrary
> workloads out of the box.
> 
> Now, what about the other problems that have been reported in this
> thread by Ivan and others?  I don't think that it can be said that the
> only problem that ZFS has is with memory.  Unfortunately, it looks like
> these "other" problems aren't well quantified, so I think that they are
> being unfairly dismissed.  But at the same time, maybe these other
> problems are rare and unique enough that they represent very special
> cases that won't be encountered by most people.  But it also tells me
> that ZFS is still immature, at least in FreeBSD.
> 
> The universal need for tuning combined with the poorly understood
> problem reports tells me that administrators considering ZFS should
> expect to spend a fair amount of timing testing and tuning.  Don't
> expect it to work out of the box for your situation.  That's not to
> say that it's useless; there are certainly many people who can attest to
> it working well for them.  Just be prepared to spend time and possibly
> money making it work, and be willing to provide good problem reports for
> any non-memory related problems that you encounter.

To be clear, in this thread I have been mostly restricting myself to 
discussion of kmem problems only, although I have also noted that there 
are known ZFS bugs including bugs that are unfixed even in solaris (the 
ZIL low memory deadlock is one of them).  Indeed, pjd has a long list of 
bug reports from me :)

I agree with the rest of this summary.

Kris
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"


From: Scott Long <scottl@...>
Subject: Re: ZFS honesty
Date: Jan 6, 5:54 pm 2008

Kris Kennaway wrote:
> Scott Long wrote:
>> Kris Kennaway wrote:
>>> Ivan Voras wrote:
>>>> Kris Kennaway wrote:
>>>>> Ivan Voras wrote:
>>>>>> Robert Watson wrote:
>>>>>>
>>>>>>> I'm not sure if anyone has mentioned this yet in the thread, but 
>>>>>>> another thing worth taking into account in considering the 
>>>>>>> stability of ZFS is whether or not Sun considers it a production 
>>>>>>> feature in Solaris.  Last I heard, it was still considered an 
>>>>>>> experimental feature there as well.
>>>>>>
>>>>>> Last I heard, rsync didn't crash Solaris on ZFS :)
>>>>>
>>>>> [Citation needed]
>>>>
>>>> I can't provide citation about a thing that doesn't happen - you 
>>>> don't hear things like "oh and yesterday I ran rsync on my Solaris 
>>>> with ZFS and *it didn't crash*!" often.
>>>>
>>>> But, with some grains of salt taken, consider this Google results:
>>>>
>>>> * searching for "rsync crash solaris zfs": 790 results, most of them 
>>>> obviously irrelevant
>>>> * searching for "rsync crash freebsd zfs": 10,800 results; a small 
>>>> number of the results is from this thread, some are duplicates, but 
>>>> it's a large number in any case.
>>>>
>>>> I feel that the number of Solaris+ZFS installations worldwide is 
>>>> larger than that of FreeBSD+ZFS and they've had ZFS longer.
>>>
>>> Almost all Solaris systems are 64 bit.
>>>
>>> Kris
>>
>> So, let's be honest here.  ZFS is simply unreliable on FreeBSD/i386.
>> There are things that you can do mitigate the problems, and in certain
>> well controlled environments you might be able to make it work well
>> enough for your needs.  But as a general rule, don't expect it to work
>> reliably, period.  This is backed up by Sun's own recommendation to not
>> run it on 32-bit Solaris.
>>
>> But let's also be honest about ZFS in the 64-bit world.  There is ample
>> evidence that ZFS basically wants to grow unbounded in proportion to the
>> workload that you give it.  Indeed, even Sun recommends basically
>> throwing more RAM at most problems.  Again, tuning is often needed, and
>> I think it's fair to say that it can't be expected to work on arbitrary
>> workloads out of the box.
>>
>> Now, what about the other problems that have been reported in this
>> thread by Ivan and others?  I don't think that it can be said that the
>> only problem that ZFS has is with memory.  Unfortunately, it looks like
>> these "other" problems aren't well quantified, so I think that they are
>> being unfairly dismissed.  But at the same time, maybe these other
>> problems are rare and unique enough that they represent very special
>> cases that won't be encountered by most people.  But it also tells me
>> that ZFS is still immature, at least in FreeBSD.
>>
>> The universal need for tuning combined with the poorly understood
>> problem reports tells me that administrators considering ZFS should
>> expect to spend a fair amount of timing testing and tuning.  Don't
>> expect it to work out of the box for your situation.  That's not to
>> say that it's useless; there are certainly many people who can attest to
>> it working well for them.  Just be prepared to spend time and possibly
>> money making it work, and be willing to provide good problem reports for
>> any non-memory related problems that you encounter.
> 
> To be clear, in this thread I have been mostly restricting myself to 
> discussion of kmem problems only, although I have also noted that there 
> are known ZFS bugs including bugs that are unfixed even in solaris (the 
> ZIL low memory deadlock is one of them).  Indeed, pjd has a long list of 
> bug reports from me :)
> 
> I agree with the rest of this summary.
> 
> Kris

I guess what makes me mad about ZFS is that it's all-or-nothing; either
it works, or it crashes.  It doesn't automatically recognize limits and
make adjustments or sacrifices when it reaches those limits, it just
crashes.  Wanting multiple gigabytes of RAM for caching in order to
optimize performance is great, but crashing when it doesn't get those
multiple gigabytes of RAM is not so great, and it leaves a bad taste in
my mouth about ZFS in general.

Scott
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

From: Pawel Jakub Dawidek <pjd@...>
Subject: Re: When will ZFS become stable?
Date: Jan 7, 5:59 am 2008

On Fri, Jan 04, 2008 at 12:42:28PM +0100, Ivan Voras wrote:
> Hi,
>=20
> As far as I know about the details of implementation and what would it
> take to fix the problems, is it safe to assume ZFS will never become
> stable during 7.x lifetime?

To sum up this thread, let me present ZFS status as of today.

Before I do that, one explanation. I was away from FreeBSD for like 3-4
weeks, because of real life issues, etc. I hope, I'm now back for good.
Let me also use this again to invite any interested committers to help
working on ZFS (I'm inviting people to help from a day one).

Ok.

The most pressing issues currently are:
1. kmem_map exhaustion.
2. Low memory deadlocks in ZFS itself.

I believe 2nd problem is already fixed in OpenSolaris, at least that was
my impression when I made last integration, I need to double check. If
that's true, I'll try to commit the fix before 7.0-RELEASE.

The 1st problem has of course much wider audience. First of all you
need:

	http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch

The patch is not yet committed, because I was discussing better
solutions with alc@. I don't think we (he) will be able to come up with
something better before 7.0-RELEASE, so I'm going to ask re@ for
approval for this patch today. Note that it is low risk change, because
it is executed only in situation where the system will panic anyway.

Of course it is so much better to use ZFS on 64bit systems, but it also
works on i386. I'm running ZFS in production for many months on two i386
systems. One has 1GB memory and those tunning in loader.conf:

vfs.zfs.prefetch_disable=3D1
vm.kmem_size=3D671088640
vm.kmem_size_max=3D671088640

I've three ZFS pools in here, no UFS at all. The load is rather light,
serving large files. No panics.

The second "production" box is my laptop. I've 2GB of RAM (it worked
fine with 1GB too), but I do have 'options KVA_PAGES=3D512' in my kernel
config and my loader.conf looks like this:

vm.kmem_size=3D1073741824
vm.kmem_size_max=3D1073741824
vfs.zfs.prefetch_disable=3D1

My laptop is ZFS-only. No panics whatsoever.

The box I'm running ZFS for the longest time is amd64 system with 1GB of
RAM. This box is used for backups (ZFS snapshots are so damn handy) and
guess what, I'm using rsync for backups:) It also serves files through
NFS:

beast:root:~# showmount -e | wc -l
      31

ZFS is used heavly here:

beast:root:~# zfs list -t filesystem | wc -l
      50
beast:root:~# zfs list -t snapshot | wc -l
    1029

And loader.conf:

vm.kmem_size=3D629145600
vm.kmem_size_max=3D629145600

And again, rock stable.

All my ZFS systems use vm_kern.c.2.patch.

Of course all this doesn't mean ZFS works great on FreeBSD. No. It is
still an experimental feature. I don't agree we should deny mounting ZFS
on i386, etc. We can improve warning and even advise increasing
KVA_PAGES on i386. It's too late to increase vm.kmem_size by default, as
it can affect other parts of the system. ZFS also can't do it
automatically.

In my opinion people are panicing in this thread much more than ZFS:)
Let try to think how we can warn people clearly about proper tunning and
what proper tunning actually means. I think we should advise increasing
KVA_PAGES on i386 and not only vm.kmem_size. We could also warn that
running ZFS on 32bit systems is not generally recommended. Any other
suggestions?

--=20
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!