FreeNAS

I'm with access to Greg's box to analysis what is the reason for this crash.

Marcelo Araujo wrote:

I'm with access to Greg's box to analysis what is the reason for this crash.

Hi Greg,

Spent all day looking into your machine, I could see the vm crash with SIGBUS. While I was investigating this issue I noticed you have lots of errors like this one:

sonewconn: pcb 0xfffff8013195f570: Listen queue overflow: 151 already in queue awaiting acceptance (4 occurrences)

A total of:
root@freenas:~ # dmesg | grep sonewconn | wc -l
3419

It can be caused by your Broadcom NIC, however I'm still not 100% sure about that.

What I did few hours ago was set this sysctl: sysctl kern.ipc.soacceptqueue=4096

By default the value is 128 that is pretty low for some expressive network traffic, I did up this value to 4096 and restarted some services such like: mdnsd and netatalk as well as your 3 vms.

What I would suggest for you would be to set this systctl at System->Tunables and reboot your FreeNAS.

Launch again your 3 vms and let me know if that solves the VM crashes.
I'm running your 3 vms for over 3 hours already without a crash.

Best,

Marcelo Araujo wrote:

Marcelo Araujo wrote:

I'm with access to Greg's box to analysis what is the reason for this crash.

Hi Greg,

Spent all day looking into your machine, I could see the vm crash with SIGBUS. While I was investigating this issue I noticed you have lots of errors like this one:

sonewconn: pcb 0xfffff8013195f570: Listen queue overflow: 151 already in queue awaiting acceptance (4 occurrences)

A total of:
root@freenas:~ # dmesg | grep sonewconn | wc -l
3419

It can be caused by your Broadcom NIC, however I'm still not 100% sure about that.

What I did few hours ago was set this sysctl: sysctl kern.ipc.soacceptqueue=4096

By default the value is 128 that is pretty low for some expressive network traffic, I did up this value to 4096 and restarted some services such like: mdnsd and netatalk as well as your 3 vms.

What I would suggest for you would be to set this systctl at System->Tunables and reboot your FreeNAS.

Launch again your 3 vms and let me know if that solves the VM crashes.
I'm running your 3 vms for over 3 hours already without a crash.

Best,

Thank you for spending the time debugging this. I set the sysctl value and rebooted. I'll let you know if they crash again.

Greg Fitzgerald wrote:

I was up all night working, I woke up at 2:30PM EST and my VM's had crashed again. I attached the debug.tgz.

Hello Greg,

I have connected in your machine and the kern.ipc.soacceptqueue is still 128.

root@freenas:~ # sysctl kern.ipc.soacceptqueue
kern.ipc.soacceptqueue: 128

Did you roll it back?

Yes, when I setup the sysctl in the gui I had it set as loader instead of sysctl. When I booted back up I failed to verify that it was set correctly. I have since fixed it, VMs are still crashing regularly.

The fix has increased my network throughput on NFS shares by a lot though. Not sure the exact numbers. More than doubled. I wonder if this option could be set with the autotune when a realtek card is detected in the system?

PR: https://github.com/freenas/os/pull/133
and https://github.com/freenas/ports/pull/125