Project

General

Profile

Bug #28431

Possible Memory Leak in smbd

Added by Ian Cartwright over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
No priority
Assignee:
Timur Bakeyev
Category:
OS
Target version:
Seen in:
Severity:
Reason for Closing:
Third Party to Resolve
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

A week or so ago my FreeNAS server stopped responding and needed a forced reboot. I don't remember the exact error message but basically it was that no more bytes could be allocated to a process. All of my RAM and swap had been used up. Looking back at the Memory reports once the box came back up, yes "Wired" RAM went to 100% and a few days later swap did too. I didn't notice any performance degradation, though this is just a home file server and I wasn't paying a whole lot of attention to it until it stopped responding. Since the reboot I've been keeping closer tabs on memory usage and have notice that the smbd process is taking up more and more memory. Right now, it's up to 3.1GB. The only two changes I've made to the box over the last couple of weeks are upgrading from 11.0-U2 to 11.1-RELEASE and removing the box from an AD domain (basically removing all the config options from the Directory Service > Active Directory section and disabling it since it's not possible to remove the config entirely).

There is a thread over on the forums that describes just the problem I'm having, but I can't see where Morpheus187 opened a ticket, so I'm opening one of my own: https://forums.freenas.org/index.php?threads/samba-using-up-most-of-the-ram.61039


Related issues

Related to FreeNAS - Bug #28027: Samba using up all available ramClosed2018-01-28
Related to FreeNAS - Bug #27200: Possible SMB Memory LeakClosed2017-12-11
Related to FreeNAS - Umbrella #28585: Fix Samba memory leak in vfswrap_getwd()Done

History

#1 Updated by Dru Lavigne over 2 years ago

  • Private changed from No to Yes
  • Reason for Blocked set to Need additional information

Ian: please attach a debug (System -> Advanced -> Save Debug).

#2 Updated by Ian Cartwright over 2 years ago

  • File debug-nas1-20180212072934.tgz added

Attaching debug as requested.

#3 Updated by Dru Lavigne over 2 years ago

Ian: can you also update to 11.1 and let us know whether that resolves the issue? There were some fixes in that update to address RAM exhaustion.

#4 Updated by Ian Cartwright over 2 years ago

Updated to 11.1-U1 this morning. I'll keep an eye on memory usage and update with any changes. It took a couple of weeks for smbd to grow to 3GB last time, so it might take a while to get there.

#5 Updated by Dru Lavigne over 2 years ago

Thanks, Ian!

#6 Updated by Ian Cartwright over 2 years ago

Status: smbd is now up to 646MB RAM.

#7 Updated by Dru Lavigne over 2 years ago

Ian: attach a new debug (U1) and we'll take a look at what's happening.

#8 Updated by Ian Cartwright over 2 years ago

  • File debug-nas1-20180215082459.tgz added

Updated debug attached.

#9 Updated by Dru Lavigne over 2 years ago

  • File deleted (debug-nas1-20180212072934.tgz)

#10 Updated by Dru Lavigne over 2 years ago

  • Category set to OS
  • Assignee changed from Release Council to Alexander Motin
  • Target version set to 11.2-RC2

Alexander: do you see any likely culprits here?

#11 Updated by Alexander Motin over 2 years ago

  • Assignee changed from Alexander Motin to Timur Bakeyev

It may be duplicate of another smbd memory leak issue report few days ago. Timur, could you please take a look.

#12 Updated by Andrew Walker over 2 years ago

  • Related to Bug #28027: Samba using up all available ram added

#13 Updated by Andrew Walker over 2 years ago

  • Related to Bug #27200: Possible SMB Memory Leak added

#14 Updated by Timur Bakeyev over 2 years ago

  • Related to Umbrella #28585: Fix Samba memory leak in vfswrap_getwd() added

#15 Updated by Timur Bakeyev over 2 years ago

  • Status changed from Not Started to In Progress
  • Seen in changed from 11.1 to 11.1-U1

#16 Updated by Timur Bakeyev over 2 years ago

Sorry, Ian, we didn't forget about you, just trying to get some idea how to reproduce this memory leak.

Maybe you can describe, what sort of tasks you was performing that lead to increase of the memory usage by smbd process?

#17 Updated by Ian Cartwright over 2 years ago

I can't think of anything special I am doing. It's a typical home file server... Here's what is normally accessing FreeNAS via Samba at any time:

- Plex running on its own Linux VM (Ubuntu 16.04 LTS)
- Transmission running on its own Linux VM (Ubuntu 16.04 LTS)
- Crashplan running on a Mac (High Sierra)
- SecuritySpy running on that same Mac for camera captures
- The occasional file access from about 5 Mac and Windows clients

The only thing that comes even close to anything I might consider heavy usage is when Crashplan scans a few thousand files that it is backing up to the cloud for changes. And maybe Plex library updates, but it's been months or years since the last time I did a full library scan.

#18 Updated by Ian Cartwright over 2 years ago

Some more information:

The smbd process that is taking up all the memory is owned by a user that I only use for Crashplan and my camera capture files (the Mac names "spy" in the list above). Overnight there are about 70k "opened file/closed file" event pairs (140k log lines). This happens every night as Crashplan scans the files I am backing up to their cloud.

#19 Updated by Timur Bakeyev over 2 years ago

Hi, Ian!

Andrew Walker seems found at least one cause of uncontrolled memory leakage of smbd process.

Can you try the following configuration changes under Services->SMB:
1) Uncheck Unix Extensions
2) Add auxiliary parameter wide links = yes

Can you, please, try this settings and check if that helps with your problem?

#20 Updated by Ian Cartwright over 2 years ago

Sure. Before I change the settings, what affect will disabling UNIX Extensions have on my samba clients? Should I expect my Mac and Linux clients to have issues, at least temporarily?

Thanks.

#21 Updated by Timur Bakeyev over 2 years ago

Since you mentioned Linux clients...

       unix extensions (G)

           This boolean parameter controls whether Samba implements the CIFS
           UNIX extensions, as defined by HP. These extensions enable Samba to
           better serve UNIX CIFS clients by supporting features such as
           symbolic links, hard links, etc... These extensions require a
           similarly enabled client, and are of no current use to Windows
           clients.

           Note if this parameter is turned on, the wide links parameter will
           automatically be disabled.

           See the parameter allow insecure wide links if you wish to change
           this coupling between the two parameters.

           Default: unix extensions = yes

So, you may "lose" files on the shares that were actually symlinks to some other files. Please keep in mind, that Unix extensions work only with SMB 1.x protocol, which is widely disabled due security reasons on most deployments.

Alternatively, I think, you can just put it auxiliary parameters:

allow insecure wide links = yes
wide links = yes

And leave unix extensions on. We believe that issue is connected with the wide links = no setting.

#22 Updated by Ian Cartwright over 2 years ago

Thanks for the details. I've made the changes you originally asked for (UNIX Extensions off and wide links = yes). I'll keep an eye on memory usage.

#23 Updated by Ian Cartwright over 2 years ago

It's been 4 days an none of the smbd processes has grown to more than 190M according to top. The process at 190M is owned by the Crashplan users (this is the process that would grow to multiple GB before). The other smbd processes are around 150M.

The config changes seem to have done the trick.

Thanks.

#24 Updated by Timur Bakeyev over 2 years ago

  • Status changed from In Progress to Closed
  • Reason for Closing set to Third Party to Resolve

That's a good news! I'm closing this ticket and going to send updates to the Umbrella ticket regarding the final fix.

#25 Updated by Timur Bakeyev over 2 years ago

  • File deleted (debug-nas1-20180215082459.tgz)

#26 Updated by Timur Bakeyev over 2 years ago

  • Private changed from Yes to No

#27 Updated by Dru Lavigne over 2 years ago

  • Target version changed from 11.2-RC2 to N/A
  • Reason for Blocked deleted (Need additional information)

Also available in: Atom PDF