Project

General

Profile

Bug #77646

Add to isp(4) tunables to limit MSI/MSI-X usage for older cards

Added by Adrian Carpenter almost 3 years ago. Updated almost 3 years ago.

Status:
Done
Priority:
No priority
Assignee:
Alexander Motin
Category:
OS
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

I have a lab setup which uses Fibre channel to provide boot disks and xen storage. All the FC are 4GB Qlogic, and they connect to a Qlogic SANbox 5602. The setup worked like a dream for several years without issue with FreeNAS 9.xx running. I have just upgraded to 11.2 and this breaks my system. LUNS are provided and can be seen on the QLogic BIOS but as soon as a machine starts to boot some or all the paths to the LUNS disappear from Switch fabric, and the machine fails to boot. I believe all the fibre cards have their most recent firmware and aslo the switch.

Screenshot 2019-03-01 09.50.13.png (291 KB) Screenshot 2019-03-01 09.50.13.png Adrian Carpenter, 03/01/2019 05:49 AM
Screen Shot 2019-03-03 at 14.47.38.png (140 KB) Screen Shot 2019-03-03 at 14.47.38.png Adrian Carpenter, 03/03/2019 06:48 AM
Screen Shot 2019-03-03 at 18.00.55.png (123 KB) Screen Shot 2019-03-03 at 18.00.55.png tuneable Adrian Carpenter, 03/03/2019 10:18 AM
Screen Shot 2019-03-03 at 18.18.49.png (136 KB) Screen Shot 2019-03-03 at 18.18.49.png two nodes booted Adrian Carpenter, 03/03/2019 10:19 AM
Screen Shot 2019-03-03 at 18.26.57.png (157 KB) Screen Shot 2019-03-03 at 18.26.57.png before nodes have booted Adrian Carpenter, 03/03/2019 10:27 AM
57207
57783
57828
57837
57846

Associated revisions

Revision 02e06893 (diff)
Added by Alexander Motin almost 3 years ago

Add to isp(4) tunables to limit MSI/MSI-X usage. There are some problem reports possibly related to the new driver use of multiple interrupts on older cards. Hopefully this allow to workaround them. MFC after: 1 week Sponsored by: iXsystems, Inc. Ticket: #77646 (cherry picked from commit b62270da2c55c6c2184ed55804b1a342974492ae)

Revision ab73e2d2 (diff)
Added by Alexander Motin almost 3 years ago

Limit 24xx adapters to only one MSI-X interrupt by default. These are 4Gb/s and pretty old and slow now, so I see no reason to fight for their performance over stability. PR: 233654 MFC after: 1 week Sponsored by: iXsystems, Inc. Ticket: #77646 (cherry picked from commit d85065741ec7c23aff8cd48656f1791905c8bae8)

Revision 140f6225 (diff)
Added by Alexander Motin almost 3 years ago

Add to isp(4) tunables to limit MSI/MSI-X usage. There are some problem reports possibly related to the new driver use of multiple interrupts on older cards. Hopefully this allow to workaround them. MFC after: 1 week Sponsored by: iXsystems, Inc. Ticket: #77646 (cherry picked from commit b62270da2c55c6c2184ed55804b1a342974492ae) (cherry picked from commit 02e06893c97952112bc527c49e062dde7e6eface)

Revision 8f572fdd (diff)
Added by Alexander Motin almost 3 years ago

Limit 24xx adapters to only one MSI-X interrupt by default. These are 4Gb/s and pretty old and slow now, so I see no reason to fight for their performance over stability. PR: 233654 MFC after: 1 week Sponsored by: iXsystems, Inc. Ticket: #77646 (cherry picked from commit d85065741ec7c23aff8cd48656f1791905c8bae8) (cherry picked from commit ab73e2d2eee67bcfb291610f22d50bbc7169d819)

Revision c4065e45 (diff)
Added by Alexander Motin almost 3 years ago

Limit 24xx adapters to only MSI interrupts by default. This was actually the known good configuration we used before. Single MSI-X configuration doesn't even work there on my tests, just due to lack of documentation not sure whether by design or I am doing something wrong. PR: 233654 MFC after: 1 week Ticket: #77646

Revision 5cc7a6e6 (diff)
Added by Alexander Motin almost 3 years ago

Limit 24xx adapters to only MSI interrupts by default. This was actually the known good configuration we used before. Single MSI-X configuration doesn't even work there on my tests, just due to lack of documentation not sure whether by design or I am doing something wrong. PR: 233654 MFC after: 1 week Ticket: #77646 (cherry picked from commit c4065e45c1119f767599c422406fb122615ed395)

History

#1 Updated by Adrian Carpenter almost 3 years ago

  • File debug-yoda-20190227161323.txz added
  • Private changed from No to Yes

#2 Updated by Dru Lavigne almost 3 years ago

  • Assignee changed from Release Council to Alexander Motin

#3 Updated by Alexander Motin almost 3 years ago

  • Status changed from Unscreened to In Progress
  • Target version changed from Backlog to 11.2-U3

According to FreeBSD PR 233654 this may be related to the new driver attempt to use multiple MSI-X interrupts when possible, that for some reason cause problems on old 24xx adapters. I've made a patch to disable this functionality on 24xx adapters and also provided control over it with loader tunables for any later needs.

#4 Updated by Alexander Motin almost 3 years ago

  • Status changed from In Progress to Ready for Testing

I've merged it to 11.2-stable: 02e06893c97952112bc527c49e062dde7e6eface ab73e2d2eee67bcfb291610f22d50bbc7169d819 5cc7a6e623b0d6679f3791398351d83aa36cbf69

I've tested that patch does what it should, but can't test whether it really help, since I never saw that problem myself. Adrian, could you try to experiment with next nightly build and let us know whether it help?

#5 Updated by Adrian Carpenter almost 3 years ago

Yes I will, when will the nightly be available in UK?

#6 Updated by Alexander Motin almost 3 years ago

They are normally built every night on US time, so I'd say tomorrow afternoon should work. You should be able to check for the change by looking into `man isp`, which should there document `hint.isp.N.msi` loader tunables.

#7 Updated by Adrian Carpenter almost 3 years ago

57207

I updated and the attached screen shot shows which version I am on. I don't see anything in man isp relating to hint.isp.N.msi.

BW

Adrian

#8 Updated by Dru Lavigne almost 3 years ago

  • File deleted (debug-yoda-20190227161323.txz)

#9 Updated by Dru Lavigne almost 3 years ago

  • Subject changed from FC in Target mode disconnects to Add to isp(4) tunables to limit MSI/MSI-X usage for older cards
  • Private changed from Yes to No
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#11 Updated by Alexander Motin almost 3 years ago

Adrian Carpenter wrote:

I updated and the attached screen shot shows which version I am on. I don't see anything in man isp relating to hint.isp.N.msi.

The screenshot tells you are still on 0227 with chance to update to 0228. To be sure I'd recommend you to wait for 0301.

#12 Updated by Adrian Carpenter almost 3 years ago

Thanks, I'll leave it for a couple of days.

#13 Updated by Adrian Carpenter almost 3 years ago

57783

Hi,

I have the patched version, and with hint.isp.X.role = 1 and hint.isp.X.msix my switch is now showing the ports as "unknown"

do I need to set hint.isp.X.msi ?

Adrian

#14 Updated by Alexander Motin almost 3 years ago

Could you show what exactly have you set? You shouldn't need to set msix, since it should be zero by default for 24xx chips. msi should also be set to 1, same as it was before. You should not set role=1 explicitly, you should enable FC ports in CTL and let it do things right.

#15 Updated by Adrian Carpenter almost 3 years ago

57828
57837
57846

It appears to be working!

Two Xen have booted successfully

Setup as in attached.

One thing curious switch shows ports as in second screenshot. Before the second node booted my switch repaired the vendor as FreeNAS ISCSI

#16 Updated by Alexander Motin almost 3 years ago

Adrian Carpenter wrote:

It appears to be working!

I'd say it works despite your attempts to break it:
- There are no msx parameters, either msi or msix, and setting msix to 1 would be a failure, while msi is already defaults to 1.
- As I have told, you should not set role=1 explicitly, since it will bring the port up before the port is configured by CTL. You should bring respective ports up after boot with `ctladm port -o on -t fc`.

#17 Updated by Alexander Motin almost 3 years ago

  • Status changed from Ready for Testing to Passed Testing

#18 Updated by Dru Lavigne almost 3 years ago

  • Status changed from Passed Testing to Done

Also available in: Atom PDF