Project

General

Profile

Bug #67854

Listen queue overflow errors

Added by Mike P 10 months ago. Updated 9 months ago.

Status:
Closed
Priority:
No priority
Assignee:
Sean Fagan
Category:
OS
Target version:
Severity:
New
Reason for Closing:
Cannot Reproduce
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Hi I just recently upgraded to the most recent version of FreeNAS 11.2 and have started to see alerts regarding Listen queue overflow. Being a Windows guy I have very limited knowledge of BSD but I have had a stable system for over 2 years - now I see this error so I'm a bit concerned. I would appreciate some guidance on how to start diagnosing the issue - and please treat me a like a 3yr old as I barely know how to fire up the shell ! :) Thanks.

ER-NAS.eyvo.com kernel log messages:

sonewconn: pcb 0xfffff8009c768910: Listen queue overflow: 151 already
in queue awaiting acceptance (1 occurrences)
sonewconn: pcb 0xfffff8009c768910: Listen queue overflow: 151 already
in queue awaiting acceptance (1 occurrences)


Related issues

Has duplicate FreeNAS - Bug #67861: Listen queue overflow errors - ** sorry not sure why this incident was posted twice.Closed

History

#1 Updated by Mike P 10 months ago

  • File debug-ER-NAS-20190105185418.txz added
  • Private changed from No to Yes

#2 Updated by Mike P 10 months ago

  • Subject changed from Listen queue overflow eerors to Listen queue overflow errors

#3 Updated by Dru Lavigne 9 months ago

  • Has duplicate Bug #67861: Listen queue overflow errors - ** sorry not sure why this incident was posted twice. added

#4 Updated by Dru Lavigne 9 months ago

  • Assignee changed from Release Council to Alexander Motin

#5 Updated by Alexander Motin 9 months ago

  • Assignee changed from Alexander Motin to Sean Fagan

#6 Updated by Sean Fagan 9 months ago

  • Status changed from Unscreened to Blocked
  • Reason for Blocked set to Waiting for feedback

This means there's a large number of connection requests coming in, and a process is not responding to them quickly enough. However, I don't know if there's a way (from the debug file) to map the address to a particular process. I don't see any errors from any processes, so I have no way of knowing what's causing the backlog to generate.

If the problem is continuing, please run "lsof" on the NAS, and capture the output. That may have the address in the output, to allow for correlation.

#7 Updated by Mike P 9 months ago

  • File putty.log added

Sean Fagan wrote:

This means there's a large number of connection requests coming in, and a process is not responding to them quickly enough. However, I don't know if there's a way (from the debug file) to map the address to a particular process. I don't see any errors from any processes, so I have no way of knowing what's causing the backlog to generate.

If the problem is continuing, please run "lsof" on the NAS, and capture the output. That may have the address in the output, to allow for correlation

The output of lsof is attached to this message as a file called putty.log - does this help ?

#8 Updated by Sean Fagan 9 months ago

I need to know the kernel messages from when the lsof is run. Thanks! (Just need one of them, since they all seemed to have the same address.)

#9 Updated by Mike P 9 months ago

OK Im happy to try and provide whatever you need but bearing mind I'm a LINUX newbie (in fact Im Linux ignorant) and that it took me an 40mins of research to figure out to capture the lsof output to you .... how do I capture the kernal messages ? and what is the order of commands you need done ?

I installed Putty - managed to login to the box - ran lsof - captured the lsof output into a log file
now how do I get the kernal messages to you ? Sorry I know this is stupidly low level stuff :(

#10 Updated by Sean Fagan 9 months ago

Sorry, you were so informative I thought you were more familiar.

The kernel messages can be seen by "dmesg", or in the file /var/log/messages, or /var/log/dmesg.* (there should be two of those files, one for today and one for yesterday). Running "lsof" and capturing the output via putty was exactly right; I just the kernel messages of the form "sonewconn: pcb 0xfffff8009c768910 <rest of stuff>" from the same session as the lsof output. If you haven't rebooted since running lsof, then just the ones now; otherwise, please run "dmesg ; lsof" and capture all the output.

Thanks!

#11 Updated by Mike P 9 months ago

  • File putty2-log.txt added

Lol - I'm glad I pass for someone that pseudo knows what they are doing - I'm a Windows guy so Linux is totally foreign - like Chinese - anyway here is another file with the output of dmesg - how does this look ?

#12 Updated by Sean Fagan 9 months ago

  • Status changed from Blocked to Closed
  • Reason for Closing set to Cannot Reproduce
  • Reason for Blocked deleted (Waiting for feedback)

Unfortunately, I don't see any match in lsof output for the complaint from the kernel. That means that, unfortunately, I can't tell what is misbehaving.

The result of the listen queue overflowing is that attempts to connect to that particular socket will fail. I don't see any complaints about that in the log files in the debug attachment, but I could have missed them. More likely, though, it's something being requested by another machine (or a jail or VM), which means no logging would happen on this end.

I'm afraid there isn't anything we can do about this, because we don't have enough information. Unless you've got messages from client machines, or from jails/VMs, that something is failing, I can't tell from this what is going on.

I'm very sorry about this.

#13 Updated by Dru Lavigne 9 months ago

  • File deleted (debug-ER-NAS-20190105185418.txz)

#14 Updated by Dru Lavigne 9 months ago

  • File deleted (putty.log)

#15 Updated by Dru Lavigne 9 months ago

  • File deleted (putty2-log.txt)

#16 Updated by Dru Lavigne 9 months ago

  • Target version changed from Backlog to N/A
  • Private changed from Yes to No

#17 Updated by Alexander Motin 9 months ago

Just for a note, there is ongoing parallel investigation of listen queue overflows in Samba for other customer. It should be fixed in some nearest releases.

Also available in: Atom PDF