Project

General

Profile

Bug #2293

MSI interrupts on 9.1

Added by Paul Bucher over 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Expected
Assignee:
Paul Bucher
Category:
Middleware
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

When installing [[FreeNAS]] 9.1 into a vmware ESXi VM MSI interrupts are not working which is keeps me from bringing up my VMs with multiple CPU cores without having interrupt storms. [[FreeNAS]] 8.3 & 8.2 have worked great for me in VMs and I've also tested [[FreeBSD]] 9.1 with the same setup without issue.

See Ticket #1894 for some chatter on this issue.

Below are are some diffs between the dmesg buffer between the [[FreeNAS]] 9.1 box and the [[FreeBSD]] box. Both booting up a single CPU core. I'm seeing ioapic being used on the [[FreeNAS]] box and assigning a traditional interrupt and the [[FreeBSD]] box is using msi to route to a MSI interrupt #.

[[FreeNAS]] 9.1 Version:

pcib0: matched entry for 0.21.INTA
pcib0: slot 21 INTA hardwired to IRQ 18
pcib3: slot 0 INTA is routed to irq 18
mps0: <LSI SAS2308> port 0x4000-0x40ff mem 0xd2400000-0xd240ffff,0xd2440000-0xd247ffff irq 18 at device 0.0 on pci3
mps0: Firmware: 14.00.01.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: attempting to allocate 1 MSI vectors (1 supported)

pcib0: slot 22 INTA hardwired to IRQ 19
pcib11: slot 0 INTA is routed to irq 19
vmx3f0: <VMware Vmxnet3 Ethernet Controller> port 0x5000-0x500f mem 0xd2504000-0xd2504fff,0xd2503000-0xd2503fff,0xd2500000-0xd2501fff irq 19 at device 0.0 on pci11
vmx3f0: Driver version : 0.1.0.0.
vmx3f0: Capabilities : sg jf vlan csum tso
vmx3f0: attempting to allocate 1 MSI vectors (1 supported)
vmx3f0: Allocating MSI irq failed, error : 6
ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 55
vmx3f0: bpf attached

[[FreeBSD]] 9.1 Version:

vmx3f0: <VMware Vmxnet3 Ethernet Controller> port 0x4000-0x400f mem 0xd2404000-0xd2404fff,0xd2403000-0xd2403fff,0xd2400000-0xd2401fff irq 18 at device 0.0 on pci3
vmx3f0: Driver version : 0.1.0.0.
vmx3f0: Capabilities : sg jf vlan csum tso
vmx3f0: attempting to allocate 1 MSI vectors (1 supported)
msi: routing MSI IRQ 256 to local APIC 0 vector 53
vmx3f0: using IRQ 256 for MSI
vmx3f0: bpf attached

pcib0: matched entry for 0.22.INTA
pcib0: slot 22 INTA hardwired to IRQ 19
pcib11: slot 0 INTA is routed to irq 19
mps0: <LSI SAS2308> port 0x5000-0x50ff mem 0xd2500000-0xd250ffff,0xd2540000-0xd257ffff irq 19 at device 0.0 on pci11
mps0: Firmware: 14.00.01.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: attempting to allocate 1 MSI vectors (1 supported)
msi: routing MSI IRQ 257 to local APIC 0 vector 54
mps0: using IRQ 257 for MSI

Loader.conf from [[FreeBSD]] 9.1 box/FreeNAS has the same stuff in it

hw.pci.enable_msi=1
hw.pci.enable_msix=0
vmxnet3_load=YES

vmstat -i on [[FreeNAS]] box

interrupt                          total       rate
irq1: atkbd0                           6          0
irq15: ata1                          285          0
irq16: vmx3f1 ehci0                11085         27
irq17: mpt0                         5715         14
irq18: uhci0 mps0                  60807        153
irq19: vmx3f0                       2004          5
cpu0:timer                         51924        130
Total                             131826        332

History

#1 Updated by Cyber Jock over 8 years ago

+2. Myself and 1 other person have this issue.

Kind of disappointing that the milestone was deleted since ESXi + [[FreeNAS]] + M1015 with passthrough is the recommended configuration in the forums for using [[FreeNAS]] as a guest OS with ESXi. The recommended configuration is precisely why I bought the hardware I did.

C'est la vie.

http://forums.freenas.org/threads/absolutely-must-virtualize-freenas-a-guide-to-not-completely-losing-your-data.12714/

#2 Updated by Paul Bucher over 8 years ago

Replying to [comment:2 noobsauce80]:

Kind of disappointing that the milestone was deleted since ESXi + [[FreeNAS]] + M1015 with passthrough is the recommended configuration in the forums for using [[FreeNAS]] as a guest OS with ESXi. The recommended configuration is precisely why I bought the hardware I did.

Agreed. I've got a huge investment in [[FreeNAS]] Virtual SANs(60+ SAS drives spread over 5 servers). If it comes down to it, I'll build a version using [[FreeBSD]] 9.1-stable which I've already verified as not having this problem.

#3 Updated by Anonymous over 8 years ago

Milestones are used internally for job tracking and are not a user-serviceable knob. Deleting the milestone on a newly created ticket where it was erroneously set by the submitter is standard operating procedure and should not be interpreted as a decision related to whether your ticket will be serviced or not.

Also, tickets are not subject to voting. Please do not spam tickets with "I agree" or "+1", and keep comments on-topic.

Thank you for your understanding.

#4 Updated by Cyber Jock over 8 years ago

Replying to [comment:4 dwhite]:

Also, tickets are not subject to voting. Please do not spam tickets with "I agree" or "+1", and keep comments on-topic.

That would be news to me(and all of the mods in the forum). This has been SOP since I started on the forums. The mods of the forums have been telling people to add "+1" or "I agree" to tickets that affect them so as to help the project managers determine which tickets have an impact on the most users.

If this policy has changed recently (or didn't change and someone got bad info on feedback on tickets) then this should be mentioned in the forum's admin section so that all the mods are aware of this. Otherwise you can expect to see more "+1" posts in the future.

#5 Updated by William Grzybowski over 8 years ago

[[FreeNAS]] 9.1 runs [[FreeBSD]] 9-stable, so thats not an excuse.

#6 Updated by Paul Bucher over 8 years ago

Opps I'm a [[FreeBSD]] noob,

I meant to say 9.1-release. So yes something appears broken between 9.1-release and 9-stable.

#7 Updated by Cyber Jock over 8 years ago

If [[FreeBSD]] stable is broken while [[FreeBSD]] release isn't, then shouldn't a bug report be files with [[FreeBSD]] stable to get it fixed and then let the fix trickle down to [[FreeNAS]]?

I'm asking the question because I wasn't too familiar as to the difference between RELEASE, CURRENT, and STABLE until 10 minutes ago...

I can't imagine that pbucher and I are the only 2 that know of this issue, so I'd imagine that at ticket exists somewhere for this already. But I'm not sure where to look or where to post to get the bug fixed.

#8 Updated by Paul Bucher over 8 years ago

Ok I pulled down the 9.1-stable snapshot built on 6/23/13 and verified that MSI interrupts are broken on it, so it is indeed an upstream problem. I'm afraid I'm a bit lost on where to start nudging upstream to get this fixed.

I'm guessing since several IX employes are also contributors to [[FreeBSD]] then someone here can maybe point us in the right direction.

I agree with noobsauce that surely someone else has hit this bug, but a quick search of the [[FreeBSD]] forums & lists hasn't yielded me much.

#9 Updated by Paul Bucher over 8 years ago

Solved:

Please Please remove these lines from /sys/dev/pci/pci.c

/*
237     * MSI-X allocation doesn't work properly for devices passed through
238     * by VMware up to at least ESXi 5.1.
239     */
240     { 0x079015ad, PCI_QUIRK_DISABLE_MSI, 0, 0 }, /* PCI/PCI-X */
241     { 0x07a015ad, PCI_QUIRK_DISABLE_MSI, 0, 0 }, /* PCIe */

Next:

Showing my [[FreeBSD]] nob status, does any know how to get ahold of marius the author of the revision 247632 to [[FreeBSD]]? Disabling MSI-X is correct, it's just taking out MSI also is killing all of us. I'd like to contact him and see if he's willing to do some work to only disable MSI-X interrupts instead of both. I'm not seeing the ability offhand to black list only MSIX interrupts in the code.

Work Around:

Remove all passed through cards and only 1 CPU for doing updates and installs and then adding the tuning constant "hw.pci.honor_msi_blacklist=0"

#10 Updated by Anonymous over 8 years ago

should get to him, though if you want to talk about PCI stuff you will want to involve John Baldwin () as well.

#11 Updated by Anonymous over 8 years ago

r247632 is against HEAD, not stable/9. That quirk doesn't exist in [[FreeNAS]]. Therefore disabling the blacklist should have no effect in this situation.

There is a global tunable "hw.msi.msix_enable" that would force the system to MSI only if the problem is that just MSI-X is broken. Similarly there is a "hw.msi.msi_enable" for MSI but that may include MSI-X by extension.

More research is needed here to pin down exactly where things are falling down.

#12 Updated by Paul Bucher over 8 years ago

Trust me ;)

I've done the research, though I've not seen hw.msi before(might you mean hw.pci.enable_msi). I'm very familiar with pci.enable_msi and that's how I make this whole thing fly, see ticket 2056.

Also the revision to stable is 248052 for merging in 247632 from head.

#13 Updated by Paul Bucher over 8 years ago

Thanks to the contact info, I tweaked the pci.c code and sent it to both of them to see if they'll put it into the [[FreeBSD]] source repo. I'll update this once I hear back from them. This fix should also close ticket 2056.

#14 Updated by Anonymous over 8 years ago

r247632 isn't in [[FreeNAS]] either. Here is the file that is in the source tree used for the 9.1 builds:

https://github.com/trueos/trueos/blob/freenas-9-stable/sys/dev/pci/pci.c

I don't see the quirk there, unless I'm missing something.

#15 Updated by Paul Bucher over 8 years ago

Sure it is ;)

Here's a patch for pci.c based on what I sent the [[FreeBSD]] maintainers and your git repo(which is just my changes). I haven't tried compiling it but I have faith that barring a typo it should seal the deal on all of this.

Edit:

Removed the patch. The maintainers have a different one that does the same thing.

#16 Updated by Paul Bucher over 8 years ago

Ok, John Baldwin got back to me with a patch that does something very similar to mien, which solves this issue. I added a 2nd patch to his that would fix the issue in ticket 2056. I built [[FreeNAS]] using releng/9.1.0 as of last night and have deployed my patched beta build to 3 of my servers now with good results.

I'll update this ticket again once something gets committed to the [[FreeBSD]] repository.

#17 Updated by Paul Bucher over 8 years ago

Ok the above mentioned patch didn't work out, so I went back to the drawing board and finally nailed down all the issues and have successfully tested them in several different environments.

I've forwarded John Baldwin a tweak to the patch he sent me which will enable [[FreeBSD]]/FreeNAS/etc to simply work out the box with ESXi VMs that have passed through PCI cards that use MSI interrupts(including the very popular LSI cards). This also allows for multiple CPU cores in the VM also without any kind of tunning tweaks applied.

#18 Updated by Anonymous over 8 years ago

Please post the [[FreeBSD]] svn rev number once that commit has been made so we can merge the change. Thanks!

#19 Updated by Paul Bucher over 8 years ago

Ok the rev # for 9.1 stable is 253273 which is MFC r253120. I've been running [[FreeNAS]] releng/9.1.0 from 2 weeks ago with this patch applied to it for the last 2 in several of my non-mission cirtical vSANs with good results. Also this will close ticket# 2056.

#20 Updated by William Grzybowski over 8 years ago

Committed as [1bf3929c] in [[TrueOS]].

Thanks

#22 Updated by Paul Bucher over 8 years ago

Thanks, you guys rock.

#23 Updated by Cyber Jock over 8 years ago

Outstanding! You guys are amazing! I will finally be able to use [[FreeNAS]]9 on my ESXi server. Woohoo!

Thanks pbucher for providing the assistance to the [[FreeNAS]] developers.

#24 Updated by Jordan Hubbard about 8 years ago

  • Assignee changed from William Grzybowski to Paul Bucher
  • Seen in set to 9.1.0-RELEASE

Just checking - this DID get resolved? Or not? Following the update change is kind of confusing, and I still see the quirks in question in FreeBSD 9-stable.

#25 Updated by Paul Bucher about 8 years ago

Jordan Hubbard wrote:

Just checking - this DID get resolved? Or not? Following the update change is kind of confusing, and I still see the quirks in question in FreeBSD 9-stable.

Yes it was resolved and is working great. I've got 2 really large FreeNAS 9.1.1 servers fully virtualized that get pounded on a daily basis, plus a few other ones that see lighter usage.

The quirks are valid, it just required some enhancement to the code to not black list both MSI and MSI-X interrupts via the quicks, but to allow the option to black listing of MSI-X only. The code was included in FreeBSD 9.2 release and William pulled the MFC into TrueOS just in the nick of time to make the FreeNAS 9.1 release.

Now the bigger question is did vmware get their act together and fix MSI-X interrupts in ESXi 5.5. I've got 5.5 server up and running but I haven't taken the time to see if MSI-X interrupts work now.

Thanks for checking in, you probably got more back then what you where looking for.

Also available in: Atom PDF