Hi,
In light of the recent discussion about ZFS stability, does anyone still =
have kmem_map_too_small panics on AMD64 after tuning it (as presented in =
http://wiki.freebsd.org/ZFSTuningGuide)?
I'm interested in reports like this:=20

http://kerneltrap.org/mailarchive/freebsd-current/2007/9/21/271557 (note =
that this report is for an untuned system).

Hi
No crashes so far. ftp2.ch.freebsd.org is running FreeBSD 7 Beta4 with

ZFS on a 64bit Intel Quad Core with 4GB since 2-3 months. We provide a

cvsup mirror with cvsup.ch.freebsd.org and a portsnap mirror with

portsnap3.freebsd.org too.
As an official mirror for kde, mysql, fedora, opensuse, openoffice we

run several rsync processes a day and offer rsync for mirroring. No

problem so far. I never had any ZFS related crash.
I just set two paramters in the loader.conf:

vm.kmem_size_max="1073741824"

vm.kmem_size="1073741824"

and kern.maxvnodes=400000 in sysctl.conf
I often talk to Solaris admins for larg server farms. They normally use

at least 1 GB ram per 1 TB disk space with ZFS. I did the same thing.
Regards,

Thomas

_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

I have my Music and movies collection on a ZFS pool for more than a month

now which is shared through samba over the network with an XBox

MediaCenter without any problem, hitches or disconnects.
No tuning. Everything out of the box.
The zfs pool is jailed and within the jail runs samba.
The only "issue" I have is that whenever I reboot that I have to re-jail

the zfs pool and remount it within the jail. Would love if this could be

automatic.
Details:
FreeBSD hulk.superhero.nl 7.0-RC1 FreeBSD 7.0-RC1 #0: Mon Dec 24 18:52:08

CET 2007     admin@hulk.superhero.nl:/usr/obj/usr/src/sys/GENERIC  amd64
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2511.41-MHz K8-class

CPU) Origin = "AuthenticAMD"  Id = 0x60fb1  Stepping = 1

Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>

Features2=0x2001<SSE3,CX16>

AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>

AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch>

Cores per package: 2

usable memory = 3526770688 (3363 MB) ** actual 4 GB but I have memory hole

mapping disabled as enabling renders the system unusable due to being

unable to mount / ***

avail memory  = 3411177472 (3253 MB)
last pid: 95799;  load averages:  0.00,  0.00,  0.00   up 13+03:50:10

23:06:09

64 processes:  1 running, 63 sleeping

CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

Mem: 107M Active, 478M Inact, 626M Wired, 216K Cache, 214M Buf, 2050M Free

Swap: 4096M Total, 4096M Free
hulk# sysctl -a | grep kmem

vm.kmem_size_scale: 3

vm.kmem_size_max: 419430400

vm.kmem_size_min: 0

vm.kmem_size: 419430400
hulk# zpool status

  pool: zfspublic

 state: ONLINE

 scrub: none requested

config:
        NAME        STATE     READ WRITE CKSUM

        zfspublic   ONLINE       0     0     0

          raidz1    ONLINE       0     0     0

            ad4     ONLINE       0     0     0

            ad6     ONLINE     ...
[ message continues ]

Would having the option to specify the JID one wants for a jail that's

launching solve this ?
Regards,
Hugo

_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

After rebooting I have to do the following on the box.
On the host:

hulk# zfs jail 4 zfspublic/batman
This allows the jail with id4 (called batman) to talk to the zfs pool.
On the jail:
batman# zfs mount -a
Only after that the zfs pool/storage is available from within the jail.
I did have a quick look to see if I could add something to the rc.zfs file

but it's a bit tricky, depending on when the zfs is started/loaded and

when the jails are initialised. You can only jail zfs when the jail in

which you want to use zfs is already running. Please correct me if I am

wrong.
Fortunately I do not reboot that often ;-)
_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

I have 3 amd64 systems & an i386 system (workstation) using ZFS; I've

seen panics related to ZFS only on the i386 machine as expected. (Not to

mention it's still -CURRENT from months ago :))
The amd64 servers are a home server (with jails hosted on ZFS), and two

servers in Canada. Of these two, one is also a jail server (one of the

jails is a mail server that'll be fairly loaded when it goes live) and

the other is a backup server (still not in production).
The most I've thrown at ZFS so far is buildworld & ports, so perhaps

that's why I have not seen any problems yet. I'll inform the list if I

encounter any such panics when the systems go live, but for the time

being, I just wanted to say "no problems on amd64 here".
As a sidenote, I use ganglia and I wrote a little script to export the

arcsize sysctl to ganglia, from what I've seen it likes to go up til the

limit, but will relinquish memory when there's pressure; I can see such

ups and downs easily in the graphs if I stress another part of the system.
Regards,
Hugo

_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

I have vm.kmem_size="1073741824"

I used to have vm.kmem_size_max=1073741824 as well but heard it was not

needed to have both. (it made no change)

I never see my vfs.numvnodes over 60,000 but i have the limit at 400,000

anyways
amd64 freebsd

FreeBSD 7.0-BETA4 #0: Sat Dec 15 06:40:09 EST 2007

    graff@nfsd.com:/usr/obj/usr/src/sys/NFSD

Timecounter "i8254" frequency 1193182 Hz quality 0

CPU: Dual-Core AMD Opteron(tm) Processor 2218 (2593.52-MHz K8-class CPU)

  Origin = "AuthenticAMD"  Id = 0x40f12  Stepping = 2
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>

  Features2=0x2001<SSE3,CX16>

  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>

  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>

  Cores per package: 2

usable memory = 8580689920 (8183 MB)

avail memory  = 8257769472 (7875 MB)

ACPI APIC Table: <091107 APIC1523>

FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
All this server does is run nfsd, no apache, no mysql, no rsync, no zfs

snapshots either.
Jan  1 15:48:09 nfsd kernel: ZFS filesystem version 6

Jan  1 15:48:09 nfsd kernel: ZFS storage pool version 6

Jan  1 15:48:09 nfsd savecore: reboot after panic: kmem_malloc(131072):

kmem_map too small: 864546816 total allocated
In fact i was going to paste the zpool status here, but typing it in

caused the server to crash. In short i have 4 drives paired in 2

mirrors, for a raid 0 + 1 effect, normal SATA drives directly connected

to zfs.

I have tested this hardware thoroughly without zfs, memory test, make

buildworld tests, etc so i know it is stable hardware.
I hate to report bad news but i hope this leads to a fix
_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

Yes, I've had one such crash.
The machine has 4GB of RAM and initially I set it up with vm.kmem_size

and vm.kmem_size_max set to 1024M. It's my home fileserver and I'm the

only regular user. After booting the box I had tested the transfer speed

back and forth over the net and after that not really used it any more

that day. The next day I discovered that it had crashed sometime during

the night. I've since changed the two tuneables above to 1536M and I've

not had any crashes since. I also tried with 2048M but that only

resulted in a crash upon boot.
Kind regards
Morten Strårup
_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

Not sure if you wanted negatives, but I have not seen such crashes on any o=

f=20

the three amd64 systems I'm running ZFS on (3 gb, 4 gb and 4 gb of RAM). Th=

e=20

only tuning done is to increase the kmem size and arc_max and disabling=20

prefetch. The increase in kmem was not to avoid crashes, but was to=20

accomodate the larger arc_max chosen.
=2D-=20

/ Peter Schuller
PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller@infidyne.com>'

Key retrieval: Send an E-Mail to getpgpkey@scode.org

E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org

ny of=20
This is somewhat a special case - you also did tuning besides kmem size. =
I'm especially interested in this, hoping to gather the "combination=20

that works" from the reports.

I can tell you what works for us. Slightly verbose and repeating things

most of us already know, for the archives.
Do not put your swap on ZFS.
On an amd64 system with 2GB of RAM, we put the following in

/boot/loader.conf:

vfs.zfs.arc_max="600M"

vm.kmem_size_max="1G"

vm.kmem_size="1G"
We seem to pretty easily panic our RELENG_7 ZFS systems if we do not set

arc_max to roughly 65% or less of kmem_size_max. I do 60% to be more

conservative. YMMV, but you get the idea.
To panic our systems quickly when the tuning settings are bad (in other

words, to trigger a kmem_map_too_small panic) we can use iozone. It

usually only takes a minute or two.
--

TerraNovaNet Internet Services - Key Largo, FL

Voice: (305)453-4011 x101   Fax: (305)451-5991

http://www.terranova.net/

----------------------------------------------

Life's not fair, but the root password helps.

_______________________________________________

freebsd-current@freebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-current

To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"