Project

General

Profile

Bug #28471

increase memory use without explanation

Added by enrique campo over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Nice to have
Assignee:
Timur Bakeyev
Category:
OS
Target version:
Seen in:
Severity:
Reason for Closing:
Reason for Blocked:
Need additional information from Author
Needs QA:
No
Needs Doc:
No
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Hello, after updating my FreeNAS to 11.1-U1

The use of the memeoria has increased to the point of using swap, stopping the system, to use it again I must restart everything.

I do not have a previous copy to return

1.jpg (275 KB) 1.jpg enrique campo, 02/13/2018 08:33 AM
2.jpg (333 KB) 2.jpg enrique campo, 02/13/2018 08:33 AM
3.jpg (267 KB) 3.jpg enrique campo, 02/13/2018 08:33 AM
4.jpg (237 KB) 4.jpg enrique campo, 02/13/2018 08:35 AM
5.jpg (122 KB) 5.jpg enrique campo, 02/13/2018 09:35 AM
6.jpg (276 KB) 6.jpg enrique campo, 02/13/2018 09:35 AM
7 datail storage.jpg (206 KB) 7 datail storage.jpg enrique campo, 02/13/2018 12:03 PM
8.jpg (269 KB) 8.jpg enrique campo, 02/13/2018 12:44 PM
9.jpg (195 KB) 9.jpg enrique campo, 02/14/2018 04:04 AM
10.jpg (272 KB) 10.jpg enrique campo, 02/14/2018 04:32 AM
11.jpg (268 KB) 11.jpg enrique campo, 02/14/2018 09:39 AM
fre1.jpg (200 KB) fre1.jpg enrique campo, 03/22/2018 04:48 AM
fre2.png (99.4 KB) fre2.png enrique campo, 03/22/2018 04:51 AM
fre3.png (116 KB) fre3.png enrique campo, 03/22/2018 08:20 AM
fre-ult.png (122 KB) fre-ult.png enrique campo, 04/18/2018 06:36 AM
14345
14346
14347
14348
14351
14352
14355
14359
14367
14368
14372
15519
15522
15531
16764

Related issues

Related to FreeNAS - Umbrella #28585: Fix Samba memory leak in vfswrap_getwd()Done

History

#1 Updated by enrique campo over 2 years ago

  • File debug-SRV-PMV-FREE-20180213163249.txz added
  • Private changed from No to Yes

#2 Updated by enrique campo over 2 years ago

14345

#3 Updated by enrique campo over 2 years ago

14346

#4 Updated by enrique campo over 2 years ago

14347

#5 Updated by enrique campo over 2 years ago

14348

#6 Updated by enrique campo over 2 years ago

14351
14352

#7 Updated by Dru Lavigne over 2 years ago

  • Seen in changed from 11.1-U1 to 11.1-U1
  • Reason for Blocked set to Need additional information

Where is /shares/backup/FreeNAS? It has run out of space.

#8 Updated by enrique campo over 2 years ago

my freenas is installed in USB memory 8GB, with storage raidz2 of 4 hard disk

mainly used for Windows SMB

#9 Updated by enrique campo over 2 years ago

14355

#10 Updated by Dru Lavigne over 2 years ago

I still don't understand where /shares/backup/FreeNAS is located. Is it on a backup system? It has run out of space.

#11 Updated by enrique campo over 2 years ago

shares backup FreeNAS is located in an external unit,

These backups are from windows SMB

#12 Updated by enrique campo over 2 years ago

14359

#13 Updated by enrique campo over 2 years ago

Today it has been necessary to restart at least 4 times but it happens the same
Can you assure me what to do?

#14 Updated by Dru Lavigne over 2 years ago

  • Assignee changed from Release Council to Alexander Motin
  • Reason for Blocked changed from Need additional information to Need verification

Alexander: do you see any other obvious hardware reasons for the reboots. He is using a Realtek NIC and that USB backup device is problematic, but I'm not sure if that is related to the memory spike.

#15 Updated by Alexander Motin over 2 years ago

  • Assignee changed from Alexander Motin to Timur Bakeyev

According to the screenshots it is Samba process consumes most of system's memory. It should not be that huge.

Timur, could you take a look please?

#16 Updated by enrique campo over 2 years ago

14367

the restart that I mention, I have done it.

if I do not restart I receive> swap_pager_getswapspace (32): failed

how it happened during the night

#17 Updated by enrique campo over 2 years ago

14368

This is overwhelming, it happens around every hour.

memory runs out and start using swap

#18 Updated by Alexander Motin over 2 years ago

enrique, we understood that your smbd is leaking, but it would be good if you tell something about your environment and workload: what is special in your setup that could trigger the issue, what most other users supposedly don't have. Also try to correlate that growth with some specific kinds of activity.

#19 Updated by enrique campo over 2 years ago

thanks alenxerder for your word and guidance.

my freenas was installed for the year 2015 with version 9.2.1.8, in 8 GB usb.
We created a raidz2 storage with 4 hard drive of 1TB C / U
clearly to share, store information with windows smb, snapshot is active, this for approximately 50 users
mediaplex was installed in that same year, but its use was eventually to play institutional videos locally on the network

SMB windows are backed up, outside of freenas with rsync

until then only the available updates are made

I do not feel that the cresimiento or demand of the system, is greater, on the contrary, today they are less the user around 20

However, the hardware has been strengthened to 16 GB ram, at the beginning it was 4 GB, I changed to 8 GB and a few days ago I upgraded to 16 GB, because I assumed that it required more memory for what is happening

Excuse me for my English

some other information about my freenas, what to report, to know what happens, I will do it pleasantly

#20 Updated by enrique campo over 2 years ago

something I forget to mention, yesterday that this happened, I had only turned off and turned on the computer had no users using the system, today, if they are connected, I have had to restart freenas on two occasions, for now. My fear is that it be constant.

until the new version is solved,

#21 Updated by enrique campo over 2 years ago

I started a report a few days ago, but I did not know how to explain what happened, I do not know if it could be useful

https://redmine.ixsystems.com/issues/28172#change-163649

#22 Updated by Timur Bakeyev over 2 years ago

Hi, Enrique!

Can you describe what kind of operations do you perform with SMB? Any specific pattern that leads to the memory overuse?

I.e. you are backing up a lot of files from or to Windows machine? Your directory structure contains a lot of files in on directory?

#23 Updated by Timur Bakeyev over 2 years ago

  • Status changed from Not Started to In Progress

Enrique, I understand that your current experience with 11.1-U1 is frustrating, but you always have an option to roll back to the previous boot environment ( BE ). Of course we'd rather try to resolve your issue.

Meanwhile, can you monitor the memory usage of the smbd process - as I understand it grows up within an hours, so it's relatively easy to predict when it'll happen - and at that point run the following commands:

# top -b -osize

That should give you PID of the largest smbd process. Then
# smbcontrol PID pool-usage > usage.log

And attach usage.log file to the ticket.
# smbstatus
# smbstatus -N -B

And also include this output into the ticket.

#24 Updated by Timur Bakeyev over 2 years ago

Is there a possibility to get access to your system via TeamViewer session or by any other means?

#25 Updated by enrique campo over 2 years ago

Hello Timur! a pleasure!

currently there will be about 300GB in SMB windows, copying out of freenas,

and another 200GB in another SMB windows

but these backups are differential, so in reality it does not move large amount.

#26 Updated by enrique campo over 2 years ago

14372

after the two reboots earlier you have kept a constant use

#28 Updated by enrique campo over 2 years ago

I can attach the log with the use of smb, as soon as the increase is generated, right now it's fine

#29 Updated by Timur Bakeyev over 2 years ago

Hi, Enrigue!

I've connected over TV, but at the moment memory usage is pretty stable. What I can see is that you have pretty busy box - 34 smbd processed in the air.

Well, let's wait until you'll hit high memory usage again.

Do you have any chat client so we can discuss my actions online? Or, maybe, TV voice chat?

#30 Updated by enrique campo over 2 years ago

thanks timur, very kind

My spoken English is very bad.

if as I mentioned it stabilized, but it is very strange ago approx. 7 days had happened the same.

I'll be attentive to introduce myself again, to let you know

In a moment I retire until tomorrow, it's time to leave my job

#31 Updated by Timur Bakeyev over 2 years ago

  • Related to Umbrella #28585: Fix Samba memory leak in vfswrap_getwd() added

#32 Updated by Dru Lavigne over 2 years ago

  • Target version set to 11.2-RC2

#33 Updated by enrique campo over 2 years ago

hello timur, today I experience these events again, in a moment I attach the report that I request

#34 Updated by enrique campo over 2 years ago

  • File usage.log added
  • File usagesmbstatus.log added
  • File usagesmbstatus2.log added

attached, something else that may require let me know

#35 Updated by Timur Bakeyev over 2 years ago

Hi, Enrique!

Andrew Walker seems found at least one cause of uncontrolled memory leakage of smbd process.

Can you try the following configuration changes under Services->SMB:
1) Uncheck Unix Extensions
2) Add auxiliary parameter wide links = yes

Can you, please, try this settings and check if that helps?

#36 Updated by enrique campo over 2 years ago

hello timur!
clear with pleasure, earlier receive this information and it is already applied,
It remains to observe their behavior

#37 Updated by enrique campo over 2 years ago

hi timur

I received this notification. Is it due to the change? should I worry?

SRV-PMV-FREE.VIAJES-INDIGO.COM kernel log messages:

sonewconn: pcb 0xfffff8010273d960: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)

-- End of security output --

#38 Updated by Timur Bakeyev over 2 years ago

enrique campo wrote:

SRV-PMV-FREE.VIAJES-INDIGO.COM kernel log messages:

sonewconn: pcb 0xfffff8010273d960: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)

That's not immediately connected with the change.

But in general this is a warning indication that process can't coop with the number of incoming connections in timed fashion. From the quote it's not clear, which process faced that problem, of course Samba is most likely candidate here.

If you see this message only occasionally - you can ignore it, clients may reconnect later with no problem. If those messages are frequent - something has to be done... Server tuning, hardware upgrade... We can discuss that if it'll be frequent.

#39 Updated by Timur Bakeyev over 2 years ago

  • File deleted (debug-SRV-PMV-FREE-20180213163249.txz)

#40 Updated by Timur Bakeyev over 2 years ago

  • File deleted (usagesmbstatus2.log)

#41 Updated by Timur Bakeyev over 2 years ago

  • File deleted (usage.log)

#42 Updated by Timur Bakeyev over 2 years ago

  • File deleted (usagesmbstatus.log)

#43 Updated by enrique campo over 2 years ago

15519
15522

hello timur
Yesterday evening my freenas presented memory loss, but I could not verify what services consumed it, I was not allowed to connect remotely by ssh

#44 Updated by enrique campo over 2 years ago

15531

#45 Updated by Timur Bakeyev over 2 years ago

Hi, Enrique!

Did you managed to get access to the system and check was that exactly the Samba, which used again all the memory?

Meanwhile we seems found the memory leak and the fix is going to be included into the 11.1-U5. Not sure if there is an easy way to test that fix before the next release, unless you'll switch to nightlies. But those could be broken at any random point in time, so isn't recommended for the production system.

#46 Updated by Timur Bakeyev over 2 years ago

  • Target version changed from 11.2-RC2 to 11.1-U5

#47 Updated by Timur Bakeyev over 2 years ago

  • Status changed from In Progress to Closed
  • Reason for Blocked deleted (Need verification)
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No
  • QA Status deleted (Not Tested)

I didn't get any feedback on this ticket and the fix is coming with the next release of the FreeNAS. So, closing this ticket for now.

#48 Updated by Dru Lavigne over 2 years ago

  • Target version changed from 11.1-U5 to N/A
  • Private changed from Yes to No
  • Reason for Blocked set to Need additional information from Author

#49 Updated by enrique campo over 2 years ago

16764

hello timur, Dru excuse my absence, I had commitments outside the office ...

today the increase of momoria was presented, I feel that it was triggered when using the option of Shadow copies, had to recover the information of last week and I realized that the increase began, I presume that if what is tried to recover is a lot of data 6gb, the memory runs out and use swap and collapse.

I hope it's not too late, for your research and revision, if I can help more, I will do so, thank you for the effort in carrying and keeping this project forward.

#50 Updated by Dru Lavigne over 2 years ago

Enrique: this should be fixed for both 11.1-U5 and 11.2-BETA1 when they come out (both due in a few weeks).

#51 Updated by enrique campo over 2 years ago

thank you very much Dru, I am aware of it, I just wanted to be that observation

Also available in: Atom PDF