Tweaking ZFS on FreeBSD 8.2-RELEASE on i386

Having followed the http://wiki.freebsd.org/ZFSTuningGuide, when configuring ZFS, I was aware of the delicate nature of the kernel settings for ZFS on i386.  I recently upgraded my server to 4GB ECC from 2GB non-ECC and thought I’d like to take advantage of the extra ram, so I thought I’d play around with these options.

My current kernel config could not be simpler, ZFS-GENERIC:

include    GENERIC
ident    ZFS-GENERIC
options    KVA_PAGES=512

For this configuration I successfully used in /boot/loader.conf:

#Working options for ZFS-GENERIC 2GB RAM, KVA_PAGES=512
vm.kmem_size=”1536M”
vm.kmem_size_max=”1536M”
vfs.zfs.arc_max=”786M”

I thought it would be as simple as:

#Trial options for ZFS-GENERIC 4GB RAM, KVA_PAGES=512
vm.kmem_size=”2G”
vm.kmem_size_max=”2G”
vfs.zfs.arc_max=”1792M”

But, but to my chagrin, my system responded on boot up with a:

panic: kmem_suballoc: bad status return of 3

I was able to get around these failed settings after rebooting by using the loader prompt at boot-up.

Since it was so early in the boot up stage, and that it was kmem_suballoc, I figured there weren’t enough KVA_PAGES assigned to the kernel.  I tried a more conservative:

#Trial options for ZFS-GENERIC 4GB RAM, KVA_PAGES=512
vm.kmem_size=”1792M”
vm.kmem_size_max=”1792M”
vfs.zfs.arc_max=”1536M”

with the same result.

Not  to be discouraged, I updated my kernel config to:

include    GENERIC
ident    ZFS-GENERIC
options    KVA_PAGES=640

and set my loader.conf to:

#Trial options for ZFS-GENERIC 4GB RAM, KVA_PAGES=640
vm.kmem_size=”2G”
vm.kmem_size_max=”2G”
vfs.zfs.arc_max=”1792M”

This time the kernel booted, but after loading the (zfs) root filesystem, the system could not fork, and put me into single user mode.  I couldn’t reboot cleanly, as I couldn’t fork the shutdown command, and I was on a serial terminal, so couldn’t give it the three finger salute.

This time I tried:

#Trial options for ZFS-GENERIC 4GB RAM, KVA_PAGES=640
vm.kmem_size=”1792M”
vm.kmem_size_max=”1792M”
vfs.zfs.arc_max=”1536M”

and the system booted up fine.  I’ll have to see how this affects my stability and performance longer term.