Project

General

Profile

Bug #17690

FreeNAS 9.10+ seems to swap unnecessarily

Added by Stuart Espey about 4 years ago. Updated about 4 years ago.

Status:
Closed: Third party to resolve
Priority:
No priority
Assignee:
Josh Paetzel
Category:
OS
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

SuperMicro X10-SRi-F
Xeon E5-1650v4
2x16GB ECC DDR4 2400
8x 4TB Seagate NAS HD (RaidZ2)
2x 16GB Sandisk Cruiser Fit (Boot Mirror)
1000W PSU
ServeRAID M1115 HBA with P20 IT firmware
connected via gigabit

ChangeLog Required:
No

Description

I've been contributing to an active issue on the forum
https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/

I've seen this in both 9.10 and 9.10.1

The symptom is that systems with ostensibly plenty of ram, seem to build up a few megs to few hundred megs of swap over a short period (a matter of days).

The problem is that if a disk fails or is hot swapped, then the system will crash.

The issue is that this might be a deeper issue than just "you need more ram"

I first saw this issue when I had 8GB. Since then I've upgraded to a new system with 32GB, and I'm seeing the same amount of swap usage.

Others are too.

I've written a script which swapsoff/on the disks at regular intervals which seems to minimize the major concern, ie crashing during a disk failure, but I'm concerned there is something deeper at play in the way ZFS and BSD are interacting.

In this post:
https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-2#post-315008

I began researching into ZFS sysctl's which might be affecting it

And came across this FreeBSD bug report
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594

I see that Steven Hartland's patches seem to have been incorporated, but I can also see that Karl's still maintaining his patches.

In this FreeBSD bug they talk about having arc_free_target == v_free_target, but in the default system as running with 9.10.1, that is no longer the case. I'm wondering if this is the regression?

  1. sysctl vfs.zfs.arc_free_target vm.v_free_target
    vfs.zfs.arc_free_target: 56375
    vm.v_free_target: 173129

That seems to imply that ARC is going to try to allocate past the VM system's free target... constantly forcing the VM to find more space... eventually running out of options... thus the arc will grow to the point where it will evict pages, when it shouldn't.

230MB ARC Free Target vs
700MB VM free target.

Eventually that will lead to the VM having to shuttle 300-400MB to disk.

Also, in Karl's posts, he identifies UMA as being the cause of the issues he is seeing. I note that UMA is now enabled for zfs by default.

  1. sysctl vfs.zfs | grep uma
    vfs.zfs.zio.use_uma: 1

In the bug report, he says UMA will use up a lot of RAM as Inactive if you have a spiky load, for example, SMB, AFP, Rsync etc with lots of small files. Its a fundamental design flaw in UMA.

I started dissecting karl's bug report at this post:

https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-2#post-315008

I found a number of possibly related bugs in this bug tracker, listed in this post:
https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-3#post-315065

History

#1 Updated by Bonnie Follweiler about 4 years ago

  • Assignee set to Kris Moore

#2 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Assignee changed from Kris Moore to Josh Paetzel
  • Target version set to 9.10.1-U2

Josh, is this just a matter of us tuning down the ARC size to provide more cushion for running apps?

#3 Updated by Vaibhav Chauhan about 4 years ago

  • Status changed from Unscreened to Closed: Third party to resolve

BRB: waiting for FreeBSD patches to go in and whenever we update our base OS we will have a fix for you.

Also available in: Atom PDF