Project

General

Profile

Bug #7857

Bug #11099: NFS gets stuck if you reboot FreeNAS while NFS Server is Running

NFS gets stuck if you reboot while clients are connected - need a second reboot to fix

Added by David Glover-Aoki over 5 years ago. Updated about 4 years ago.

Status:
Closed: Duplicate
Priority:
Important
Assignee:
Josh Paetzel
Category:
OS
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

This is hard to explain so I'll jump straight into how to reproduce:

1. Enable the NFS service with default settings. Create an NFS share.
2. Connect to the NFS share from another machine. (I'm using Linux running on a Raspberry Pi but I don't know if this makes a difference.)
3. Reboot FreeNAS
4. Try to reconnect the NFS share.

It won't. "mount" hangs, even after rebooting the client.
Also at this point, if you try to stop the NFS service on FreeNAS, the "wait" indicator never goes away and the service doesn't stop.

5. Reboot FreeNAS again
6. All is now OK.

This may be related to bug #7588

History

#1 Updated by Jordan Hubbard over 5 years ago

  • Category set to 92
  • Assignee set to Josh Paetzel
  • Target version set to Unspecified

#2 Updated by Josh Paetzel over 5 years ago

  • Status changed from Unscreened to Investigation

Hrmm, I do this all the time.

When you mean reconnect the client are you saying that it doesn't automagically pick up that the server is back?

What you should see are log entries such as "The server is not responding" followed by "The server is alive"

Are you using NFSv3 or NFSv4 mounts?

#3 Updated by David Glover-Aoki over 5 years ago

I'm using NFSv3. (NFSv4 is disabled by default.)

The client doesn't auto-reconnect, no. Trying to manually reconnect hangs "mount".

I haven't checked log entries on the client but I will do so next time it happens. (It happens every time I update FreeNAS and reboot it so that probably won't take long.)

#4 Updated by Josh Paetzel over 5 years ago

You shouldn't need to remount at all. The client should report something along the lines of "the server is dead" and then when FreeNAS comes back up "the server is alive"

It's possible that trying to remount is confusing the NFS recovery behavior.

Can you attach the output of mount, nfsstat -m and df -h from the client after FreeNAS comes back up but before you try remounting?

#5 Updated by David Glover-Aoki over 5 years ago

After rebooting the fileserver, nfsstat -m on the client reveals:

# nfsstat -m
/mnt/Media from 10.0.1.8:/mnt/Data/Media
 Flags:    ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.1.8,mountvers=3,mountport=849,mountproto=udp,local_lock=none,addr=10.0.1.8

However, "df -h" hangs, as does attempting to cd into /mnt/Media

Attempting to reboot the client also hangs. After physically power cycling it, the NFS mount does not remount, and "nfsstat -m" returns nothing, presumably for the same reason that attempting to "mount" it by hand also hangs.

At this point, if you try to stop "NFS" on the FreeNAS Services page, the "in progress" indicator appears, but never disappears.

So, I reboot FreeNAS again. After reboot, NFS is stopped, presumably because I attempted to stop it before the reboot. After starting it again, and then rebooting the client one final time, the mount functions normally again.

#6 Updated by David Glover-Aoki over 5 years ago

After some further experimentation, I have found that attempting (and yet failing) to turn off the NFS service before reboot is essential to fixing the problem. If you don't attempt to turn off NFS, no amount of rebooting the server (or the client) will fix it.

Next time there's an update, I'll try pre-emptively turning off NFS before the update.

#7 Updated by David Glover-Aoki over 5 years ago

Before today's update I turned off NFS before updating, and back on after the reboot.

The client still hung, BUT was perfectly fine after a client reboot.

So this is the workaround - turn off NFS during server reboots and things are mostly OK.

#8 Updated by David Glover-Aoki over 5 years ago

Still a problem in FreeNAS-9.3-STABLE-201503270027

#9 Updated by Josh Paetzel over 5 years ago

  • Status changed from Investigation to Unscreened
  • Assignee changed from Josh Paetzel to Jordan Hubbard
  • Seen in changed from to

#10 Updated by Jordan Hubbard over 5 years ago

  • Status changed from Unscreened to Closed: Cannot reproduce

I've never seen this in the wild (or in my own NFS setup). Unable to reproduce.

#11 Updated by szczy 7 about 5 years ago

  • Seen in changed from to 9.3-STABLE-201506292332

I am running into this same issue with 9.3-STABLE-201506292332. I have run into this issue a few times now and until I stumbled upon this I was rebuilding the entire system each time.

#12 Updated by szczy 7 about 5 years ago

szczy 7 wrote:

I am running into this same issue with 9.3-STABLE-201506292332. I have run into this issue a few times now and until I stumbled upon this I was rebuilding the entire system each time.

Recreating bug using Bug #11099 --> https://bugs.freenas.org/issues/11099

#13 Updated by Xin Li about 5 years ago

  • Status changed from Closed: Cannot reproduce to Screened
  • Assignee changed from Jordan Hubbard to Xin Li

#14 Updated by Xin Li about 5 years ago

  • Related to Bug #11099: NFS gets stuck if you reboot FreeNAS while NFS Server is Running added

#15 Updated by Xin Li about 5 years ago

  • Status changed from Screened to 15

Hi,

Can you, when the NFS service become stuck, login into the console and grab the output of 'procstat -kk -a'?

#16 Updated by David Glover-Aoki about 5 years ago

  • File procstat2.txt.gz added
  • File procstat.txt.gz added

I attach two files. "procstat" is the output of that command before I try to turn off NFS. "procstat2" is after I try to turn it off, while it is hung.

#17 Updated by David Glover-Aoki almost 5 years ago

This is still "waiting for feedback" but I provided the required output two months ago and I can't change the status.

#18 Updated by Xin Li almost 5 years ago

  • Assignee changed from Xin Li to Josh Paetzel

#19 Updated by Shane Slocum almost 5 years ago

I am also seeing this issue. Whenever the FreeNAS box gets rebooted without first disconnecting NFS clients, they cannot connect when FreeNAS comes back up. I hadn't realized a second reboot would fix the issue until I saw this report. I had been troubleshooting extensively. This has happened twice for me in the past couple of months. Most recently on version 9.3-STABLE-201509022158

#20 Updated by Josh Paetzel over 4 years ago

  • Status changed from 15 to Fix In Progress
  • Priority changed from Nice to have to Important

#21 Updated by Josh Paetzel over 4 years ago

  • Status changed from Fix In Progress to Ready For Release
  • Target version changed from Unspecified to 261

#22 Updated by Vaibhav Chauhan over 4 years ago

  • Related to deleted (Bug #11099: NFS gets stuck if you reboot FreeNAS while NFS Server is Running)

#23 Updated by Vaibhav Chauhan over 4 years ago

  • Status changed from Ready For Release to Closed: Duplicate
  • Parent task set to #11099

#24 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from 261 to N/A

#25 Updated by Dru Lavigne over 2 years ago

  • File deleted (procstat2.txt.gz)

#26 Updated by Dru Lavigne over 2 years ago

  • File deleted (procstat.txt.gz)

Also available in: Atom PDF