FreeNAS 9.10+ seems to swap unnecessarily
2x16GB ECC DDR4 2400
8x 4TB Seagate NAS HD (RaidZ2)
2x 16GB Sandisk Cruiser Fit (Boot Mirror)
ServeRAID M1115 HBA with P20 IT firmware
connected via gigabit
I've been contributing to an active issue on the forum
I've seen this in both 9.10 and 9.10.1
The symptom is that systems with ostensibly plenty of ram, seem to build up a few megs to few hundred megs of swap over a short period (a matter of days).
The problem is that if a disk fails or is hot swapped, then the system will crash.
The issue is that this might be a deeper issue than just "you need more ram"
I first saw this issue when I had 8GB. Since then I've upgraded to a new system with 32GB, and I'm seeing the same amount of swap usage.
Others are too.
I've written a script which swapsoff/on the disks at regular intervals which seems to minimize the major concern, ie crashing during a disk failure, but I'm concerned there is something deeper at play in the way ZFS and BSD are interacting.
I began researching into ZFS sysctl's which might be affecting it
And came across this FreeBSD bug report
I see that Steven Hartland's patches seem to have been incorporated, but I can also see that Karl's still maintaining his patches.
In this FreeBSD bug they talk about having arc_free_target == v_free_target, but in the default system as running with 9.10.1, that is no longer the case. I'm wondering if this is the regression?
- sysctl vfs.zfs.arc_free_target vm.v_free_target
That seems to imply that ARC is going to try to allocate past the VM system's free target... constantly forcing the VM to find more space... eventually running out of options... thus the arc will grow to the point where it will evict pages, when it shouldn't.
230MB ARC Free Target vs
700MB VM free target.
Eventually that will lead to the VM having to shuttle 300-400MB to disk.
Also, in Karl's posts, he identifies UMA as being the cause of the issues he is seeing. I note that UMA is now enabled for zfs by default.
- sysctl vfs.zfs | grep uma
In the bug report, he says UMA will use up a lot of RAM as Inactive if you have a spiky load, for example, SMB, AFP, Rsync etc with lots of small files. Its a fundamental design flaw in UMA.
I started dissecting karl's bug report at this post:
I found a number of possibly related bugs in this bug tracker, listed in this post: