Project

General

Profile

Bug #15918

Memory Leak

Added by Bartosz Ciazynski over 4 years ago. Updated about 3 years ago.

Status:
Closed: User Config Issue
Priority:
No priority
Assignee:
Jordan Hubbard
Category:
Middleware
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

i5-4590, 16GB mem, 64GB Boot SSD, 4x2TB RaidZFS2,

ChangeLog Required:
No

Description

Jun 13 15:18:42 freenas swap_pager_getswapspace(16): failed

Error occurred when I performed library update of the Plex plug-in.

top.png (143 KB) top.png Bartosz Ciazynski, 07/12/2016 07:30 AM
6530

History

#1 Updated by Bartosz Ciazynski over 4 years ago

  • File debug-freenas-20160613155329.txz added

#2 Updated by Joshua Ruehlig over 4 years ago

  • Assignee changed from Joshua Ruehlig to Jordan Hubbard

Forwarding along. Not sure what this has to do with plugins?

#3 Updated by Bartosz Ciazynski over 4 years ago

You are probably right..

regards

#4 Updated by Jordan Hubbard over 4 years ago

  • Status changed from Unscreened to 15

Nothing to do with plugins, I think. This guy probably just doesn't have enough swap (or main memory) in his configuration.

Please paste in the output of:

sysctl hw.physmem
swapinfo -k

#5 Updated by Bartosz Ciazynski over 4 years ago

Hello.
Command output.

[root@freenas ~]# sysctl hw.physmem
hw.physmem: 17081950208
[root@freenas ~]# swapinfo -k
Device 1K-blocks Used Avail Capacity
/dev/ada1p1.eli 2097152 62864 2034288 3%
/dev/ada2p1.eli 2097152 64804 2032348 3%
/dev/ada3p1.eli 2097152 64232 2032920 3%
/dev/ada4p1.eli 2097152 63592 2033560 3%
Total 8388608 255492 8133116 3%
[root@freenas ~]#

regards,

#6 Updated by Jordan Hubbard over 4 years ago

  • Status changed from 15 to Closed: User Config Issue

Yeah, so, this machine is 2GB of memory which is below the FreeNAS recommended minimum of 4GB. Even the swap space is just 4GB (swap devices are mirrored, so only 1/2 the storage you see is usable) so basically, this is just a really minimal configuration even for pure storage purposes, adding plugins goes beyond the tolerable footprint and things start getting shot for lack of space (which is what that swap_pager_getswapspace() error is - it's basically trying to say "I'm out of memory and I tried to get some swap, but that's also exhausted - I give up!).

#7 Updated by Bartosz Ciazynski over 4 years ago

You talking about total memory or the memory that is left free ? (which is below 2GB), (Total is 16GB phys mem in this computer). How to extend hard drive swap space for the freenas ?

#8 Updated by Bartosz Ciazynski over 4 years ago

  • Hardware Configuration updated (diff)

configuration update

#9 Updated by Bartosz Ciazynski over 4 years ago

How can I and to which level should I enlarge (grow) the swap space ?

#10 Updated by Jordan Hubbard over 4 years ago

Swap space is generally configured at 1-2x physmem. Still, if you have 16GB of memory in this box, it's hard to say why you even need it. I can't reproduce that error when updating plex and we've had no duplicate reports, so there may be something else (use top / ktop to investigate memory usage) going haywire on your box. Impossible to say what.

#11 Updated by Bartosz Ciazynski over 4 years ago

Well I'am running one virtual machines under Virtualbox plug-in. These machine also takes some ram. In this case declared is 4GB. So 12Gb left for the rest.

last pid: 88623; load averages: 0.62, 0.45, 0.37 up 1+01:38:16 10:19:11
69 processes: 1 running, 68 sleeping
CPU: 1.6% user, 0.0% nice, 6.8% system, 0.4% interrupt, 91.3% idle
Mem: 95M Active, 444M Inact, 6737M Wired, 8288K Cache, 8574M Free
ARC: 1024M Total, 212M MFU, 567M MRU, 5024K Anon, 12M Header, 229M Other
Swap: 8192M Total, 567M Used, 7625M Free, 6% Inuse

PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
31665 bciazyns 22 20 0 4521M 4284M select 2 375:45 21.34% VBoxHeadless
2874 root 12 20 0 255M 35720K nanslp 2 2:43 0.00% collectd
5796 972 16 30 0 300M 78156K uwait 2 0:51 0.00% Plex Media Ser
6558 972 13 52 15 274M 50024K piperd 2 0:45 0.00% python
6577 972 14 48 0 240M 21044K usem 2 0:32 0.00% Plex DLNA Serv
2609 root 1 52 0 221M 15900K select 2 0:30 0.00% python2.7
2636 root 6 20 0 367M 94656K select 0 0:26 0.00% python2.7
4587 bciazyns 11 20 0 128M 1956K uwait 2 0:14 0.00% VBoxSVC
1635 root 1 20 0 14468K 1764K select 3 0:06 0.00% powerd
6042 root 6 20 0 159M 17580K select 0 0:05 0.00% python2.7
4584 bciazyns 1 20 0 68020K 744K select 2 0:04 0.00% VBoxXPCOMIPCD
1473 root 1 -52 r0 6304K 2272K nanslp 2 0:04 0.00% watchdogd
6627 www 1 20 0 193M 128K kqread 3 0:03 0.00% httpd
4560 bciazyns 12 20 0 168M 1840K select 2 0:03 0.00% vboxwebsrv
5210 www 1 20 0 193M 16K lockf 2 0:03 0.00% httpd
6632 www 1 20 0 193M 16K lockf 3 0:03 0.00% httpd
6560 root 1 20 0 77412K 4460K kqread 1 0:02 0.00% syslog-ng
1632 root 1 20 0 30260K 18104K select 3 0:02 0.00% ntpd
5209 www 1 20 0 193M 16K lockf 3 0:02 0.00% httpd
1033 root 1 20 0 71116K 19396K select 2 0:01 0.00% devd
4588 root 1 20 0 173M 368K select 2 0:01 0.00% httpd
1907 root 1 20 0 229M 3272K select 2 0:01 0.00% nmbd
56893 www 1 20 0 50500K 4500K kqread 1 0:01 0.00% nginx
5207 www 1 20 0 193M 16K lockf 1 0:01 0.00% httpd

#12 Updated by Bartosz Ciazynski over 4 years ago

  • File log.txt added

Well there are some issues with the ups connectivity but this is not a problem.

#13 Updated by Bartosz Ciazynski over 4 years ago

end of the log ---> > pid 60753 (Plex Media Scanner), uid 972, was killed: out of swap space

#14 Updated by Sean Fagan over 4 years ago

16G of memory, and 8G of swap (not mirrored, look at the output of swapinfo above).

And yet something is causing it to run out of swap space -- there are 1761 complaints about swap_pager_getswapspace failing starting at Jun 13 15:16:34. And then, about a half hour after that, it reboots. The second one shows 2732 instances of that.

We need to see the output of "swapinfo k" and "top" when that is happening - there's not enough logging information otherwise.

#15 Updated by Bartosz Ciazynski over 4 years ago

Hello,

So today my machine stalled again. Out of swap space. It's not possible to connect to it via http(s) only ssh.

[root@freenas ~]$ swapinfo -k
Device 1K-blocks Used Avail Capacity
/dev/ada0p1.eli 2097152 2097144 8 100%
/dev/ada1p1.eli 2097152 2097144 8 100%
/dev/ada2p1.eli 2097152 2097144 8 100%
/dev/ada3p1.eli 2097152 2097144 8 100%
Total 8388608 8388576 32 100%
[root@freenas ~]$

there is so little memory that I'am unable to run ktop even :) thinking about turning off the plex jail for the time being to see if the problem will persist.

#16 Updated by Bartosz Ciazynski over 4 years ago

  • Hardware Configuration updated (diff)

Hello,

Well I turned off the Plex Jail before the weekend and till now the server still works no problems whatsoever. ;)
[bciazyns@freenas] /mnt/raidZ2/bciazyns# swapinfo -k
Device 1K-blocks Used Avail Capacity
/dev/ada0p1.eli 2097152 0 2097152 0%
/dev/ada1p1.eli 2097152 0 2097152 0%
/dev/ada2p1.eli 2097152 0 2097152 0%
/dev/ada3p1.eli 2097152 0 2097152 0%
Total 8388608 0 8388608 0%
[bciazyns@freenas] /mnt/raidZ2/bciazyns#

usage of swap is 0%.

Interesting thing is that to begin with the swap error "(swap_pager_getswapspace)" began while I updated my Plex library with some large portion of pictures and video files. After that I selected from Plex to update its library and that's when the problem started. So as showed before scanner died cause it's eaten all of the memory "(Plex Media Scanner), uid 972, was killed: out of swap space".

#17 Updated by Josh Paetzel over 4 years ago

What is vfs.zfs.arc_max set to?

Sounds like it's set too high for the hardware+workload

#18 Updated by Bartosz Ciazynski over 4 years ago

Hello.

vfs.zfs.arc_max - 1073741824 - generated by the autotune.

greetings.

#19 Updated by Josh Paetzel over 4 years ago

Alright, That's 1GB, so ZFS is not causing the issue. Sounds like the machine just needs more RAM (or swap) for what it's trying to do.

#20 Updated by Bartosz Ciazynski about 4 years ago

6530

I managed to run the top command. Without memory it managed to execute :P The Plex media Scanner is definitely an issue. :) that causes the machine to stall cause of lack of swap space.

Can I enlarge the swap space to help me overwhelm the issue or that won't be enough ?

#21 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Seen in changed from Unspecified to N/A

#22 Updated by Dru Lavigne about 3 years ago

  • File deleted (debug-freenas-20160613155329.txz)

#23 Updated by Dru Lavigne about 3 years ago

  • File deleted (log.txt)

#24 Updated by Dru Lavigne about 3 years ago

  • Target version set to N/A
  • Private changed from Yes to No

Also available in: Atom PDF