Project

General

Profile

Bug #25053

Reduce RAM fragmentation

Added by Dan Brown over 1 year ago. Updated 4 months ago.

Status:
Resolved
Priority:
Important
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

In four weeks of uptime, the Inactive RAM on my FreeNAS system has steadily increased from ~700 MB to over 55 GB. My ARC has correspondingly decreased in size from over 100 GB to under 60 GB, with a hit ratio of 12-13%. I don't have 128 GB of RAM in this system, just for half of it to sit unused. What's going on here?

History

#1 Updated by Dan Brown over 1 year ago

  • File debug-freenas2-20170708074115.txz added

#2 Updated by Dru Lavigne over 1 year ago

  • Status changed from Unscreened to 15
  • Seen in changed from Unspecified to 9.10.2-U4

I'm pretty sure this was fixed with 11. Are you able to upgrade this system to confirm?

#3 Updated by Dan Brown over 1 year ago

I have a test system on 11-U1, but don't have the uptime or usage on it yet to say. But the most recent posts in this thread indicate that the problem continues in 11: https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-4

#4 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Assignee changed from Release Council to Alexander Motin

The inactive memory counter works a bit different in FreeBSD. Inactive memory is technically available / free, just hasn't been reclaimed by the kernel (yet) until it is needed.

https://unix.stackexchange.com/questions/134862/what-do-the-different-memory-counters-in-freebsd-mean

I'm unsure how this relates to your issue with ARC shrinkage, but it may (or may not). Or perhaps ARC is evicting in-active pages and those get tossed into "inactive" bucket until needed again.

Sending this over to Alexander Motin to confirm my suspicions here.

#5 Updated by Alexander Motin over 1 year ago

Inactive memory should not create pressure for ARC. Something else probably did, and inactive memory just filled the gap. My guess is that it can be a high level of kernel address space fragmentation, which does not allow ARC allocate sufficiently large blocks of kernel address space. Could you try to increase KVA size to 1.5-2x of your physical RAM (by adding loader tunable vm.kmem_size=206158430208 -- 192GB) and report whether it help?

#6 Updated by Dan Brown over 1 year ago

I haven't forgotten about this, but the increase in inactive takes place over the space of weeks. I recently upgraded my server to 11.0-U2, and will update this further once I've had a change to see what happens. Right now, the system is reported 119.4G wired, 4.0G inactive, and an ARC size of 107.8G.

#7 Updated by Dru Lavigne over 1 year ago

  • Status changed from 15 to 46

Sasha: is there anything to be done here?

#8 Updated by Alexander Motin over 1 year ago

  • Status changed from 46 to Ready For Release
  • Priority changed from No priority to Important
  • Target version set to 11.1

Still my best guess is KVA fragmentation. FreeNAS 11.1 should get recent ZFS improvement storing ARC content in 4KB chunks, that should radically reduce the fragmentation problem. Dan, if you have any more important input, please let us know, otherwise lets hope 11.1 will help.

#9 Updated by Dan Brown over 1 year ago

I'm now at 18 days uptime with 11.0-U2, and things seem to be improved, if not completely resolved. With what should be a near-identical workload, after 18 days, I'm showing 22.7G of inactive RAM, and an ARC size of 87.7G.

#10 Updated by Dru Lavigne over 1 year ago

  • File deleted (debug-freenas2-20170708074115.txz)

#11 Updated by Dru Lavigne over 1 year ago

  • Private changed from Yes to No

#12 Updated by Dru Lavigne over 1 year ago

  • Subject changed from Inactive RAM steadily increases to Reduce RAM fragmentation

#13 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.1 to 11.1-BETA1

#14 Updated by Dru Lavigne about 1 year ago

  • Status changed from Ready For Release to Resolved

#15 Updated by Joe Maloney about 1 year ago

  • Needs QA changed from Yes to No
  • QA Status deleted (Not Tested)

#16 Updated by Marco Pfeiffer 4 months ago

I see the problem in 11.1-U6, I just have 12 GB of RAM (for 9TB of storage) and my arc shrinks over about 2 days until it reaches arc_min of 1.5GB while inactive ram is about 10GB.

Restarting the NAS makes it really fast for about a day, after which the arc cache is at arc_min again.

I defined the tunable "vfs.zfs.arc_min" and set it to 8589934592 (8GB) which resolves this problem for me but I shouldn't need to do that.

#17 Updated by Dru Lavigne 4 months ago

Marco: please create a new ticket with your debug attached to it so a dev can investigate.

#18 Updated by Alexander Motin 4 months ago

ARC shrink down to arc_min indeed may be related to memory fragmentation, but there were also some reports that it can happen for other reasons already fixed in 11.2. It would be good to test that with upcoming 11.2 release or its BETA3.

Also available in: Atom PDF