Project

General

Profile

Bug #27270

Fix memory leak

Added by Richard Kojedzinszky almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Blocks Until Resolved
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

We are running a FreeNAS 11.1-STABLE, the box has 144G memory, while the ARC was at 80G, and not increasing. This seems to be the cause of nearly all memory wired, and not released to ARC.

The box had around 137G in wired state, around 4G free, and had just 80G in ARC. Running a very simple perl script, which allocates 4G memory and writes to it, resulted immeditately in 14G free memory, reduced wired memory, and with the same ARC size. After half an hour, the ARC began to increase, with wired memory also, but I suppose 86G arc memory should not need 128G wired memory.

Now this is the top header:

Mem: 91M Active, 283M Inact, 1527M Laundry, 128G Wired, 11G Free
ARC: 86G Total, 23G MFU, 55G MRU, 287M Anon, 1796M Header, 6787M Other
67G Compressed, 146G Uncompressed, 2.16:1 Ratio
Swap: 10G Total, 52M Used, 10G Free

I would better see not used wired memory as free, as in that case I am sure the ARC would be able to grow. Please find the ARC size graph from our nas, the decrease can be observed.

It could hurt performance, I suspect.

nas-a-arc-size.png (13.3 KB) nas-a-arc-size.png Richard Kojedzinszky, 12/15/2017 04:55 AM
13402

Related issues

Related to FreeNAS - Bug #27422: FreeNAS 11.1 possible memory leakClosed: Duplicate2017-12-26
Related to FreeNAS - Bug #27458: Lost swapfile after update to 11.1Closed: Insufficient Info2017-12-28
Has duplicate FreeNAS - Bug #27235: Unknown lockupClosed: Duplicate2017-12-14

History

#1 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Unscreened to 15

Richard: please attach a debug (System -> Advanced -> Save Debug).

#2 Updated by Richard Kojedzinszky almost 2 years ago

  • File debug-nas-a-20171215161259.tgz added

#3 Updated by Dru Lavigne almost 2 years ago

  • Private changed from No to Yes

Just to clarify: you're running on MASTER (nightlies) not STABLE?

#4 Updated by Richard Kojedzinszky almost 2 years ago

We are using the kernel from 11-stable branch.

#5 Updated by Dru Lavigne almost 2 years ago

  • Status changed from 15 to Unscreened
  • Assignee changed from Release Council to Alexander Motin

#6 Updated by Richard Kojedzinszky almost 2 years ago

On #27235 I've mentioned there may be a memory leak in g_bio uma zone, as we see huge numbers as used.

Please investigate. This can be reproduced easily just booting up a box, and issuing IO to a pool. In a few minutes, I got 200 entries persistently in used state, while on other FreeBSD boxes, used counts are nearly at zero.

I used kernel based on commit 881ac279c45b2d5673607a25b0b8a45fcbb876d0.

#7 Updated by Richard Kojedzinszky almost 2 years ago

Bisecting 11.1-stable and 11.0-stable for the leak, found that since commit a0dddc24c905013363838bb04e79443d81a2d765 g_bio is leaking. Hope that helps.

#8 Updated by Alexander Motin almost 2 years ago

#9 Updated by Alexander Motin almost 2 years ago

  • Subject changed from ARC and wired memory to ARC and g_bio zone memory leak
  • Status changed from Unscreened to Fix In Progress
  • Priority changed from No priority to Blocks Until Resolved
  • Target version set to 11.1-U1

Looking on FreeBSD ZFS commit log, there is a commit r323796, fixing g_bio zone memory leak in ZFS, triggered exactly by the ZFS commit you mentioned. It looks very promising. It should be in next hightly build due to massive FreeBSD merge, and I am going to merge it for 11.1-U1.

#10 Updated by Alexander Motin almost 2 years ago

I'll wait this pull request pass the tests: https://github.com/freenas/os/pull/58

#11 Updated by Alexander Motin almost 2 years ago

  • Status changed from Fix In Progress to Ready For Release

I've merged the patch into freenas/11.1-stable branch.

#12 Updated by Dru Lavigne almost 2 years ago

  • File deleted (debug-nas-a-20171215161259.tgz)

#13 Updated by Dru Lavigne almost 2 years ago

  • Private changed from Yes to No

#14 Updated by Richard Kojedzinszky almost 2 years ago

Thanks for the fast response, it seems the leak disappeared as expected. And now I see the expected high ARC sizes.

#15 Updated by Alexander Motin almost 2 years ago

  • Related to Bug #27422: FreeNAS 11.1 possible memory leak added

#16 Updated by Dru Lavigne almost 2 years ago

  • Related to Bug #27458: Lost swapfile after update to 11.1 added

#17 Updated by David Steinberger almost 2 years ago

Is there a way to try this patched kernel without changing the release train to nightly? I got the same leak but can not easily migrate back as I ported all jails over to iocage.

#18 Updated by Dru Lavigne almost 2 years ago

You can reboot and select your current boot environment from the boot menu after you test the nightly fix.

#19 Updated by Dru Lavigne almost 2 years ago

  • Subject changed from ARC and g_bio zone memory leak to Fix memory leak

#20 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Ready For Release to Resolved

Also available in: Atom PDF