Project

General

Profile

Bug #24810

Document interface name change

Added by han sooloo almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
No priority
Assignee:
Warren Block
Category:
Documentation
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Been on version 9.10.2 for a long time. Decided to upgrade to 11.

However, after the upgrade, the system does NOT automatically acquire an IP address on vmx0 interface. 'afpd' immediately starts complaining about network problems.

The only way to recover is to:
1. go to the shell and 'dhclient vmx0': it will get an IP address from the network
2. /etc/rc.d/sshd onerestart: for SSH access
3. /usr/local/etc/rc.d/nginx onerestart: for web access

I don't understand why network does NOT work at boot up, but when I manually issue 'dhclient vmx0' it does.

After manual intervention, it is possible to reach/mount ZFS shares across the network without any problems.

Associated revisions

Revision 02776280 (diff)
Added by Warren Block over 3 years ago

Note that vmx(4) drive should be used rather than VMware tools. Ticket: #24810

History

#1 Updated by han sooloo almost 4 years ago

  • File debug-ZFS-20170624222411.txz added

#2 Updated by han sooloo almost 4 years ago

Additional data points:
  1. This only happens on an upgrade. Fresh install of 11 works OK.
  2. If I restore old system's config on fresh/working install, networking fails again.
  3. During upgrade, on the console, see some errors about reflex/not-found (I think, don't recall exact messages, maybe in the logs that were uploaded?)

#3 Updated by Sean Fagan almost 4 years ago

There was a change in how upgrades were handled, and you may have needed to go to the latest 9.10.2 update first, and then upgrade to 9.11. Alternately, have done an ISO upgrade to 9.11.

#4 Updated by Sean Fagan almost 4 years ago

  • Status changed from Unscreened to 15

#5 Updated by Sean Fagan almost 4 years ago

Do you have an "update.failed" or similar file in /data?

#6 Updated by han sooloo almost 4 years ago

Sean Fagan wrote:

Do you have an "update.failed" or similar file in /data?

I could not see "/data/update.failed" or similar file.

What is curious is that the factory default factory-v1.db is much larger (~884K) than the FreeNAS 9.10.2 U2 exported settings (which were later imported) freenas-v1.db (~378K).

#7 Updated by Sean Fagan almost 4 years ago

So, as stated, I think U2 had a problem going to 11. Updating from U2 to U3 or later should have fixed that.

On the other hand, I also put in some changes to the post-upgrade scripts that should have handled it, so I'm curious what happened.

(The behaviour is that the migrations didn't apply correctly; this most likely happened because of system call incompatability.)

I'd have to see, I think, the output of "freenas-update -v -T FreeNAS-11-STABLE update" to know if that is what happened, however.

#8 Updated by han sooloo almost 4 years ago

Sean Fagan wrote:

So, as stated, I think U2 had a problem going to 11. Updating from U2 to U3 or later should have fixed that.

On the other hand, I also put in some changes to the post-upgrade scripts that should have handled it, so I'm curious what happened.

(The behaviour is that the migrations didn't apply correctly; this most likely happened because of system call incompatability.)

I'd have to see, I think, the output of "freenas-update -v -T FreeNAS-11-STABLE update" to know if that is what happened, however.

freenas-update -v -T FreeNAS-11-STABLE update

No updates available

I am wondering if there was a install/upgrade warning on the release notes for 11 that I could have missed.

Had I seen a warning message, I would not have gone to 11 directly. Although, something in the back of my mind was telling me, "dude, go from U2 to U5 first, then go to 11" ... oh well!

Last ... even a fresh 11 install is OK at first. It is ONLY when I try to restore config from 9.10.2U2 that I get into the problematic state. I was hoping that the bug upload would have provided the boat load of error messages I saw during that upgrade.

Happy to join IRC or Slack if you have real-time questions.

#9 Updated by Sean Fagan almost 4 years ago

Wait: if you do an install, and then attempt to restore the config, then it fails as well?

#10 Updated by han sooloo almost 4 years ago

Sean Fagan wrote:

Wait: if you do an install, and then attempt to restore the config, then it fails as well?

Yes

#11 Updated by Sean Fagan almost 4 years ago

  • Assignee changed from Sean Fagan to William Grzybowski

That's something other than the upgrade, then.

William, any ideas?

#12 Updated by William Grzybowski almost 4 years ago

I dont see anything wrong out of the bat... Can you please attach /var/tmp/rc.conf.freenas of the system when in bad state?

#13 Updated by William Grzybowski almost 4 years ago

sqlite3 -header /data/freenas-v1.db "select * from network_interfaces" might be useful as well.

#14 Updated by han sooloo almost 4 years ago

  • File rc.conf.freenas added
  • File network_interfaces.sqlout added

#15 Updated by William Grzybowski almost 4 years ago

Given these outputs I can see vmx0 has DHCP disabled for vmx0.

Can you go to Network - Interfaces and make sure vmx0 has DHCP enabled? Even if it has, can you try saving the form to update values?

#16 Updated by han sooloo almost 4 years ago

William Grzybowski wrote:

Given these outputs I can see vmx0 has DHCP disabled for vmx0.

Can you go to Network - Interfaces and make sure vmx0 has DHCP enabled? Even if it has, can you try saving the form to update values?

Something I just noticed ... my interfaces were named "vmxf0" and "vmxf1" in the web UI instead of what FreeBSD `ifconfig` reports as "vmx0" and "vmx1".

Did interface naming change between one of the Ux releases?

#17 Updated by William Grzybowski almost 4 years ago

han sooloo wrote:

William Grzybowski wrote:

Given these outputs I can see vmx0 has DHCP disabled for vmx0.

Can you go to Network - Interfaces and make sure vmx0 has DHCP enabled? Even if it has, can you try saving the form to update values?

Something I just noticed ... my interfaces were named "vmxf0" and "vmxf1" in the web UI instead of what FreeBSD `ifconfig` reports as "vmx0" and "vmx1".

Did interface naming change between one of the Ux releases?

You mean between 11.0 to 11.0-U1? I find that difficult.
But between 9.10.2 and 11.0, highly possible.

#18 Updated by han sooloo almost 4 years ago

William Grzybowski wrote:

han sooloo wrote:

William Grzybowski wrote:

Given these outputs I can see vmx0 has DHCP disabled for vmx0.

Can you go to Network - Interfaces and make sure vmx0 has DHCP enabled? Even if it has, can you try saving the form to update values?

Something I just noticed ... my interfaces were named "vmxf0" and "vmxf1" in the web UI instead of what FreeBSD `ifconfig` reports as "vmx0" and "vmx1".

Did interface naming change between one of the Ux releases?

You mean between 11.0 to 11.0-U1? I find that difficult.
But between 9.10.2 and 11.0, highly possible.

Yes, that was exactly the my use case where the failure happened. Upgraded using web update UI from 9.10.3 U2 to 11.0.

I, as a user, do not understand why the interface names would change behind the scenes and result in a broken network state.

#19 Updated by William Grzybowski almost 4 years ago

The log say you had vmx3f0, vmx3f1 and vmx0, it doesnt say anything about vmxf0 nor vmxf1.

#20 Updated by han sooloo almost 4 years ago

William Grzybowski wrote:

The log say you had vmx3f0, vmx3f1 and vmx0, it doesnt say anything about vmxf0.

Yes, apologies for the typo. Had 'vmx3f0' with 9.10.2 (not 9.10.3, another typo!) U2, and ended up with 'vmx0' with 11.0.

#21 Updated by William Grzybowski almost 4 years ago

  • Category changed from 1 to 129
  • Status changed from 15 to Unscreened
  • Assignee changed from William Grzybowski to Alexander Motin

Yeah, I don't know why/how it changed, lets ask our FreeBSD kernel expert.

Alexander, please see above, it seems interface name for vmx device changed from 9.10.2 to 11.0. Do you know anything about that?
I am thinking we should write a migration to follow the change but I am clueless the follow to logic, can you help?

#22 Updated by Sean Fagan almost 4 years ago

vmx3f seems to come from the VMWare tools, not from freebsd. I think we used to include the vmware tools in freenas?

#23 Updated by han sooloo almost 4 years ago

  • File 11 vmware tools.png added

Sean Fagan wrote:

vmx3f seems to come from the VMWare tools, not from freebsd. I think we used to include the vmware tools in freenas?

Not sure how, but a fresh ISO install of 11.0 on an ESXi 6.5 results in a FreeNAS VM WITH vmware tools.

#24 Updated by William Grzybowski almost 4 years ago

Sean Fagan wrote:

vmx3f seems to come from the VMWare tools, not from freebsd. I think we used to include the vmware tools in freenas?

I think vmware tools was years ago, we have been using open vm tools for quite some time.

Not sure what else we can do here, if anything. Ideas?

#25 Updated by han sooloo almost 4 years ago

William Grzybowski wrote:

Sean Fagan wrote:

vmx3f seems to come from the VMWare tools, not from freebsd. I think we used to include the vmware tools in freenas?

I think vmware tools was years ago, we have been using open vm tools for quite some time.

Not sure what else we can do here, if anything. Ideas?

I think this might be an "UPGRADE WARNING" in release notes, etc. ... When I had first installed 9.3 on ESXi 6.0, I had to manually install vmware tools to get the VMXNET3 drivers for GigE performance. But, with FreeNAS 9.10.1, apparently this is NOT needed anymore ==> https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/ (Step-9, see the SKIP THIS STEP section).

What is curious is my upgrades from 9.3 to various 9.10 flavors NEVER had a problem. I never lost my network interfaces. It was ONLY with the 9.10x to 11 upgrade that my network stopped working.

OK to close this as a "corner case, probably not supported on ESXi installs anyways" item :-)

#26 Updated by William Grzybowski almost 4 years ago

  • Category changed from 129 to Documentation
  • Assignee changed from Alexander Motin to Dru Lavigne

Dru, please refer to the comment above. Can we add something to the docs and/or release notes in that regard?

#27 Updated by Dru Lavigne almost 4 years ago

  • Status changed from Unscreened to 15

It's not in the docs or the Release Notes that I could find. Do you have a link where you're seeing it in either?

#28 Updated by Dru Lavigne almost 4 years ago

  • File deleted (debug-ZFS-20170624222411.txz)

#29 Updated by Dru Lavigne almost 4 years ago

  • File deleted (rc.conf.freenas)

#30 Updated by Dru Lavigne almost 4 years ago

  • File deleted (network_interfaces.sqlout)

#31 Updated by Dru Lavigne almost 4 years ago

  • File deleted (11 vmware tools.png)

#32 Updated by Dru Lavigne almost 4 years ago

  • Private changed from Yes to No

#33 Updated by William Grzybowski almost 4 years ago

Dru Lavigne wrote:

It's not in the docs or the Release Notes that I could find. Do you have a link where you're seeing it in either?

I didn't say it was. Thats the point. I am thinking we should add something to that matter?

#34 Updated by Dru Lavigne almost 4 years ago

  • Status changed from 15 to Closed: User Config Issue

It makes more sense to refer to that comment if it coomes up again rather than tell people not to do something. We'll watch out for queries on the forums.

#35 Updated by William Grzybowski almost 4 years ago

Dru Lavigne wrote:

It makes more sense to refer to that comment if it coomes up again rather than tell people not to do something. We'll watch out for queries on the forums.

Who is talking about telling people to not do something?

We are simply saying a message that people upgrading would be careful if they ever had VMWare tools installed.

#36 Updated by Alexander Motin almost 4 years ago

IIRC up to some point we used virtual NIC driver from VMware's official VM tools. Unfortunately those tools stopped working some time around FreeBSD 10, so we had to switch FreeNAS to open-vm-tools package instead, which implements some other functionality, but does not include NIC driver. Instead we are now using completely different vmx(4) NIC driver bundled with FreeBSD. I am not sure at what exactly point the change has happened, but indeed the interface name could change that caused this migration problem.

#37 Avatar?id=14398&size=24x24 Updated by Kris Moore over 3 years ago

  • Seen in changed from Unspecified to N/A

#38 Updated by Dru Lavigne over 3 years ago

  • Status changed from Closed: User Config Issue to Resolved
  • Target version set to 11.0-U3

#39 Updated by Dru Lavigne over 3 years ago

  • Assignee changed from Dru Lavigne to Warren Block

#40 Updated by Dru Lavigne over 3 years ago

  • Subject changed from Cannot connect to network after upgrade from 9.10.2 to 11 to Document interface name change

Also available in: Atom PDF