Project

General

Profile

Bug #4003

Windows Server 2012 R2 Failover Clustering is Failing

Added by Michael Islek about 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Nice to have
Assignee:
Josh Paetzel
Category:
OS
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Dear SIr/Madam

We have been successfully using FreeNAS with Window Server 2008 R2 to build failover clustering.Recently we have migrated to Windows Server 2012 R2 and trying to rebuild some of our clusters. However cluster validation on Windows Server 2012 R2 is falling due to inability of the server to perform persistent iSCSI-3 reservation.

In addition, although the same disk is connected by both servers as an iSCSI target, the validation reports that the iSCSI disks are not persisted on both of the cluster partner either. Consequently we can't build any Failover Clustering.

We are dead in the water. Any help that can ben provided would be greatly appreciated. We have tried this on both 9.2.0 and 9.2.1-Beta. The same results on both versions.


Related issues

Blocked by FreeNAS - Feature #4929: Generate a config file for ctldResolved2014-05-03
Blocks FreeNAS - Bug #5524: CTL should be default iSCSI in 9.3Resolved2014-07-18

History

#1 Updated by Josh Paetzel about 5 years ago

  • Status changed from Unscreened to Screened
  • Assignee set to Josh Paetzel
  • Target version changed from 9.2.0-RELEASE to 74
  • Seen in changed from 9.2.1-RELEASE to 9.2.0-RELEASE

tcpdumps of the iSCSI traffic between FreeNAS and w2k12 versus w2k8 would be useful. The target hasn't been validated with w2k12 yet, but in theory it does support SPC-3 SCSI reservations (They are used by w2k8 as well)

I'll mock this up in the lab, but it's going to be a few days.

#2 Updated by Josh Paetzel about 5 years ago

  • Subject changed from Windows Server 2021 R2 Failover Clustering is Failing to Windows Server 2012 R2 Failover Clustering is Failing

#3 Updated by Nick Morgowicz about 5 years ago

Hello, I can help provide some data if it will be of any use. SCSI-3 persistent reservations definitely do not work in Server 2012 R2, and it's clearly visible in powershell. Please take a look at this output from my Server 2012 R2 cluster:

PS C:\> Get-MPIOAvailableHW

VendorId ProductId IsMultipathed IsSPC3Supported BusType
-------- --------- ------------- --------------- -------
LSILOGIC Logical Volume False False SAS
FreeBSD iSCSI Disk True False iSCSI
SYNOLOGY iSCSI Storage True True iSCSI

I had been using Nexenta and it was fully SP3 functional before coming to FreeNAS, and every other iscsi implementation that i've tried on other operating systems hasn't been a problem, it seems as though this one is. Having SMB3 will be helpful and will work for the usecase of storing virtual machines on, so that could be one workaround until this is fixed, but it'll be nice to have a functional iscsi target in freenas.

#4 Updated by Josh Paetzel about 5 years ago

Jordan,

I've tried to email twice about this as a followup to my email on Saturday, at this point let's just make sure to talk about it tomorrow.

#5 Updated by Nick Morgowicz about 5 years ago

Josh/Jordan,

I could build up a mock lab in any version of windows and set up clustering if it could help you guys. You'd have to provide me a little more detail on which switches of which commands you'd need me to capture, but i'd be happy to help.

I'm a Microsoft Certified Professional in Server 2012 and have an MSDN account through my job, so building/deploying windows servers in my lab is cake.

#6 Updated by Nick Morgowicz about 5 years ago

  • File Failover Cluster Validation Report-20140127.0753-filedata2.mht added

Oh, to give you guys an overview of what's now looked at in clustering Server 2012 R2 machines, you can list out the cluster tests with this powershell:
PS C:\Windows\system32> Test-Cluster -List -WhatIf | ft -AutoSize

If focusing on storage only, along with descriptions and formatting in a list gives you this:
PS C:\Windows\system32> Test-Cluster -List -WhatIf | where category -eq "Storage" | fl *


Category    : Storage
DisplayName : List Disks
Description : List all disks visible to one or more nodes. If a subset of disks is specified for validation, list only disks in the subset.

Category    : Storage
DisplayName : List Disks To Be Validated
Description : List disks that will be validated for cluster compatibility.

Category    : Storage
DisplayName : Validate CSV Network Bindings
Description : Validate that network bindings required by Cluster Shared Volumes are present.

Category    : Storage
DisplayName : Validate CSV Settings
Description : Validate that settings and configuration required by Cluster Shared Volumes are present. This test can only be run with an administrative account, and it only tests servers that are cluster nodes.

Category    : Storage
DisplayName : Validate Disk Access Latency
Description : Validate acceptable latency for disk read and write operations.

Category    : Storage
DisplayName : Validate Disk Arbitration
Description : Validate that a node that owns a disk retains ownership after disk arbitration.

Category    : Storage
DisplayName : Validate Disk Failover
Description : Validate that a disk can fail over successfully with data intact.

Category    : Storage
DisplayName : Validate File System
Description : Validate that the file system on disks in shared storage is supported by failover clusters and Cluster Shared Volumes (CSVs). Failover cluster physical disk resources support NTFS, ReFS, FAT32, FAT, and RAW. Only volumes formatted as NTFS or ReFS are accessible in disks added as CSVs.

Category    : Storage
DisplayName : Validate Microsoft MPIO-based disks
Description : Validate that disks that use Microsoft Multipath I/O (MPIO) have been configured correctly.

Category    : Storage
DisplayName : Validate Multiple Arbitration
Description : Validate that in a multiple-node arbitration process, only one node obtains control.

Category    : Storage
DisplayName : Validate SCSI device Vital Product Data (VPD)
Description : Validate uniqueness of inquiry data (SCSI page 83h VPD descriptors).

Category    : Storage
DisplayName : Validate SCSI-3 Persistent Reservation
Description : Validate that storage supports the SCSI-3 Persistent Reservation commands.

Category    : Storage
DisplayName : Validate Simultaneous Failover
Description : Validate that disks can fail over simultaneously with data intact.

Category    : Storage
DisplayName : Validate Storage Spaces Persistent Reservation
Description : Validate that storage supports the SCSI-3 Persistent Reservation commands needed by Storage Spaces to support clustering.

I've run the tests on my 4 node hyper-v cluster that has 2x synology iscsi luns, 2x freenas zvol dev iscsi luns, & 2x freenas file extent on zvol iscsi luns, and only ran the test again the one file extent csv that i didnt have active data on, since i'm running ~20 virtual machines currently and don't want to stop them.

The cluster validation log is attached.

Seems as though after it fails on persistent 3 reservation check, it will stop the other tests.


Issuing call to Persistent Reservation RESERVE on Test Disk 0 from node CISSI-N1.home.morgowicz.com.
Failure issuing call to Persistent Reservation RESERVE on Test Disk 0 from node CISSI-N1.home.morgowicz.com when that node has successfully registered. It is expected to succeed. The requested resource is in use.
.
Test Disk 0 does not provide Persistent Reservations support for the mechanisms used by failover clusters. Some storage devices require specific firmware versions or settings to function properly with failover clusters. Please contact your storage administrator or storage vendor to check the configuration of the storage to allow it to function properly with failover clusters.
Failure issuing call to Persistent Reservation RESERVE on Test Disk 0 from node CISSI-N2.home.morgowicz.com when that node has successfully registered. It is expected to succeed. The requested resource is in use.
.
Stop: 1/27/2014 7:51:10 AM.
Test failed. Please look at the test log for more information.

#7 Updated by Josh Paetzel about 5 years ago

The iSCSI target in FreeNAS claims to support SPC-3 reservations. It lists w2k8r2 as supported, but not w2k12, so I'm hoping it's something small and easily fixed.

My thoughts were to run tcpdump on FreeNAS while testing validation on 2008 and 2012 to see what's different.

Assume that igb1 is the interface that you're running iSCSI over and 10.0.0.10 is the 2008 system's IP you'd do:

tcpdump -i igb1 -s 0 -w /mnt/tank/dump.out host 10.0.0.10 and port 3260

Then run the cluster validation. Ctrl-c the tcpdump when it's done. Then do the same for 2012 and attach both dumpfiles to this ticket.

If you're using MPIO kill all but one path so tcpdump gets all the traffic.

#8 Updated by Jordan Hubbard about 5 years ago

  • Target version changed from 74 to 9.2.1-RELEASE

#9 Updated by Nick Morgowicz about 5 years ago

Okay, i'll have something for you guys by Wednesday night if it's not too late.

Does it make a difference if my freenas is running an LACP lag? Or should i set up a target portal on my management interface for testing?

Also, would it make a difference if we do our testing on virtual machines running on top of a hyper-v cluster? I could serve the vm data on freenas or on the synology, and then present the clusters with iscsi from freenas. This would be the easiest for me, otherwise i could re-allocate two of my c6100 server blades to the testing, but that would take a bit more time to setup and tear down between OS testing.

#10 Updated by Josh Paetzel about 5 years ago

LACP is fine. Just use -i laggX instead of the physical interface in the tcpdump command.

The synology is interesting if you can get a packet capture from it.

Wednesday night is fine timeframe wise.

#11 Updated by Michael Islek about 5 years ago

  • File dump.out added

Gentlemen

Attached please find the tcpdump captured during a failed Validation session.

#12 Updated by Michael Islek about 5 years ago

  • File dump.out added

Josh Paetzel wrote:

The iSCSI target in FreeNAS claims to support SPC-3 reservations. It lists w2k8r2 as supported, but not w2k12, so I'm hoping it's something small and easily fixed.

My thoughts were to run tcpdump on FreeNAS while testing validation on 2008 and 2012 to see what's different.

Assume that igb1 is the interface that you're running iSCSI over and 10.0.0.10 is the 2008 system's IP you'd do:

tcpdump -i igb1 -s 0 -w /mnt/tank/dump.out host 10.0.0.10 and port 3260

Then run the cluster validation. Ctrl-c the tcpdump when it's done. Then do the same for 2012 and attach both dumpfiles to this ticket.

If you're using MPIO kill all but one path so tcpdump gets all the traffic.

Here is the TCPDUMP File captured while performing validation testing on Windows Server 2012 R2

#13 Updated by Jordan Hubbard about 5 years ago

  • Target version changed from 9.2.1-RELEASE to 72

We frankly have no idea how to fix this and may have to simply declare Win2012 as unsupported for 9.2.1. If we can find a fix later, great!

#14 Updated by Arish A almost 5 years ago

Jordan Hubbard wrote:

We frankly have no idea how to fix this and may have to simply declare Win2012 as unsupported for 9.2.1. If we can find a fix later, great!

Hello, I know that you have put this on your back burner just 3 months ago but is there any hotfix for this yet? I am new to FreeNAS and this is not turning out to be the best first experience. I would hate to move away from FreeNAS to an alternative after a whole two days of work on configuration and data migration. This is a home lab but failover clustering is still important to most people who would deploy an iscsi NAS at home.

#15 Updated by Josh Paetzel almost 5 years ago

  • Priority changed from Expected to Nice to have

We are awaiting the move to the kernel iSCSI daemon, which will likely happen this summer.

#16 Updated by Steven Sedory almost 5 years ago

After months of work and thousands of dollars spent on hardware and licensing, my mini hyperv data center project is halted.

Josh, first of all, thank you so much for what you and your team do. I rave about FreeNAS and use it in quite a few places. I know there are some other options out there for my situation, but I would of course prefer to hang onto FreeNAS. It being mid May and all, I may wait. Is there a clearer picture on when the iSCSI daemon will be moved to?

#17 Updated by Nick Morgowicz almost 5 years ago

Steven Sedory wrote:

After months of work and thousands of dollars spent on hardware and licensing, my mini hyperv data center project is halted.

Josh, first of all, thank you so much for what you and your team do. I rave about FreeNAS and use it in quite a few places. I know there are some other options out there for my situation, but I would of course prefer to hang onto FreeNAS. It being mid May and all, I may wait. Is there a clearer picture on when the iSCSI daemon will be moved to?

Steven, as an interim, you can move to an opensolaris derivative, like OmniOS. I've been using that with the free plugin, napp-it, to do my storage server, and thanks to sun's notes on converting a fiber channel hba from initiator to target mode, i got some inexpensive parts off ebay and am serving 75% of my storage traffic over 4gb FC, and the other 25% off iSCSI.

Solaris's implementation of COMSTAR allows you to do everything you need, and it supports all of the latest reservation types. There are no problems with 2012 R2 Clustering.

Nexenta released a new version of their software too, but you must have less than 18TB RAW in your pool to use theirs. They had a much prettier and usable UI than OmniOS+napp-it, but FreeNAS's is the nicest and most usable overall.

I can't wait for this support to come in-box, then i'll have to see if i can also get FC target to work before i attempt again. Best of luck.

#18 Updated by Steven Sedory almost 5 years ago

Nick Morgowicz wrote:

Steven, as an interim, you can move to an opensolaris derivative, like OmniOS. I've been using that with the free plugin, napp-it, to do my storage server, and thanks to sun's notes on converting a fiber channel hba from initiator to target mode, i got some inexpensive parts off ebay and am serving 75% of my storage traffic over 4gb FC, and the other 25% off iSCSI.

Solaris's implementation of COMSTAR allows you to do everything you need, and it supports all of the latest reservation types. There are no problems with 2012 R2 Clustering.

Nexenta released a new version of their software too, but you must have less than 18TB RAW in your pool to use theirs. They had a much prettier and usable UI than OmniOS+napp-it, but FreeNAS's is the nicest and most usable overall.

I can't wait for this support to come in-box, then i'll have to see if i can also get FC target to work before i attempt again. Best of luck.

Nick, thank you very much for the response. It gives me something to turn towards quickly, rather than searching through an endless sea of possible solutions. I could apparently use SMB3 with Microsoft's Failover Cluster Feature for the time being, but I'm really enjoying the MPIO IOPS/speeds of my iSCSI targets.

From another angle though, since these reservations work with 2008 R2 and FreeNAS, I'm wondering if there's something on the 2012 R2 side that can be adjusted as a work around. Before I try everything I can think of, do you think these avenues have already been exhausted?

I of course will be very thankful once this support comes as well. Any further advice or input is much appreciated.

#19 Updated by Steven Sedory almost 5 years ago

I just read this in a freenas forum. I'm going to give it a try. If anyone else wants to give it a try, that would be great.

kailord81, Mar 28, 2013

Re: Follow-up

jerquiaga, for Windows 2012 try this workaround - use cluster.exe command to add the disk instead of using the failover cluster console - it works for me.

Present the LUN to both nodes
On one of the node, take it online, format the disk and give it a drive letter. E.g. Q:
On the same node, install the Failover Cluster Command Interface feature.
Open command prompt, run these commands:
cluster res "res1" /create /group:"Available Storage" /type:"Physical Disk"
cluster res "res1" /priv diskpath="Q:"
cluster res "res1" /on
You can now open failover cluster console and verify the disk.

You can repeat above steps for other LUNs that you want to add - just replace "res1" with other desired descriptive name.
kailord81, Mar 28, 2013

#20 Updated by Steven Sedory almost 5 years ago

Alright, three posts in a row..Hopefully someone is reading this.

The "workaround" above seemed to render the same results, not to mention there were steps left out. The cluster must be formed first, for example.

BUT, I do have a current and apparently working cluster with two Server 2012 R2 nodes and FreeNAS iSCSI. Allow me to explain:

After not being able to make sense of the reservation differences between 2008 R2 and 2012 R2, I began to try all different variations of settings on both ends, still rendering the same results in the Verification process. Before I completely gave up and decided which FreeNAS alternative to go with, I came across this article: http://gabrewer.com/2013/01/misleading-error-in-cluster-validation-wizard/

Though the context is a bit different than ours here, the artilce reports false negatives in verification with iSCSI-3 Persistent Reservations for those upgrading from 2008 R2 to 2012(not R2). It apparently has something to do with storage spaces not being used and thus giving a "warning" (not "failure" like we're getting). Though very different from an objective standpoint, it gave me a glimpse of hope that possibly something similar was happening, especially since no one seems to be able to quantify the key differences in reservations between the two versions of Server (that I can find).

So, I went ahead and created a cluster with the iSCSI-3 Persistent Reservations failure and it's related tests being the only fault. I added four targets to the cluster, one being for a 4GB Quorum (MS says 500MB is good), and three 480GB drives to be used as CSVs. I have since created two VMs, 7 Pro and Server 2008 R2, on the same CSV. I live migrated the W7 back and forth as I was installing the OS on the Server 2008 VM. No Event errors or warnings. No apparent issues whatsoever so far.

I'm going to continue to test out this cluster, adding many more VMs to it over the next week or two. The downside of course, even if there are no problems, is that MS wont support a cluster that hasn't passed verification. I don't think I've ever called MS for support, but I'd hate to be stuck in a place where I can't if need be.

Thoughts?

#21 Updated by Anonymous almost 5 years ago

Sounds like you should call MS and verify if the validation failure is due to a bug or not, as the answer seems important to you and is worth the support incident :-)

#22 Updated by Steven Sedory almost 5 years ago

Doug White wrote:

Sounds like you should call MS and verify if the validation failure is due to a bug or not, as the answer seems important to you and is worth the support incident :-)

Thanks. I'll let you know what they say.

#23 Updated by Anonymous almost 5 years ago

Well, and also, I've been working on 2012R2 and the bug factor seems unusually high for a Windows Server release (i.e. the private/public network classification being hidden in Local Policy and not accessible from the standard UI).

#24 Updated by Nick Morgowicz almost 5 years ago

Steven, there is a difference in "making it work" and actually working.

What i ran into was due to there not being SCSI-3 reservations in FreeNAS talking correctly with Windows Server 2012 and 2012 R2, the iSCSI LUNS presented from FreeNAS will either 1) randomly go offline, 2) fail over ownership to other nodes in the cluster, 3) not appear on all nodes in the cluster when using CSV, or a combination of all of these issues. An easy way to see if things will break will be to just reboot one or more of your cluster nodes. The act of the OS trying to put a reservation on the shared storage volume is enough to cause crashes and lun outages.

It did not matter whether the disk was presented as a CSV and hosting virtual machines, or whether it was a traditional disk device in a SQL cluster. They all failed in all ways.

This bug is not with Windows, since there is no problem in using Synology iSCSI LUNs, LUNs presented from Windows Server iSCSI Target, Nexenta 3.1.5, or OmniOS. Windows can use all of those iSCSI LUNs and pass cluster validation, and can handle when cluster nodes reboot and issue reservations.

There is no workaround unless you want to share all your block storage to an OS that handles presenting shared block storage to windows and sharing it again from there; in essence, sharing twice. While you could do this if you needed, why incur the added latency and double your traffic/ops, when it's easier to use a different product that works?

#25 Updated by Steven Sedory almost 5 years ago

Nick Morgowicz wrote:

Steven, there is a difference in "making it work" and actually working.

What i ran into was due to there not being SCSI-3 reservations in FreeNAS talking correctly with Windows Server 2012 and 2012 R2, the iSCSI LUNS presented from FreeNAS will either 1) randomly go offline, 2) fail over ownership to other nodes in the cluster, 3) not appear on all nodes in the cluster when using CSV, or a combination of all of these issues. An easy way to see if things will break will be to just reboot one or more of your cluster nodes. The act of the OS trying to put a reservation on the shared storage volume is enough to cause crashes and lun outages.

It did not matter whether the disk was presented as a CSV and hosting virtual machines, or whether it was a traditional disk device in a SQL cluster. They all failed in all ways.

This bug is not with Windows, since there is no problem in using Synology iSCSI LUNs, LUNs presented from Windows Server iSCSI Target, Nexenta 3.1.5, or OmniOS. Windows can use all of those iSCSI LUNs and pass cluster validation, and can handle when cluster nodes reboot and issue reservations.

There is no workaround unless you want to share all your block storage to an OS that handles presenting shared block storage to windows and sharing it again from there; in essence, sharing twice. While you could do this if you needed, why incur the added latency and double your traffic/ops, when it's easier to use a different product that works?

Hi Nick,

That's one huge thing that I haven't done yet: a restart on one node. It was about 12 hours ago that I got it to the state it's in now. I will do a restart later to cure my optimism.

I am hesitant to go with other solutions, only because I've become so comfortable and confident in FreeNAS. Another reason is, I have a 16 Bay Supermicro server filled with 600GB 15k SAS drives, using a usb flash for FreeNAS. In my brief research, it seems this isn't safely done using Nexenta or OmniOS/Napp-it. Please correct me if I'm wrong. If not, I've been advised that mixing drive sizes/speeds on a single backplane is a bad idea, and I'd hate to use two 600GB SAS for the OS of one of these solutions, but am willing if need be. Any advice?

#26 Updated by Vladimir Usov almost 5 years ago

Please let me add some more info.
1. The Bug is easily reproducible if you run cluster validation or try to move CSV from node to node.
2. Bug is still present in 9.2.1.5
3. Bug related to Windows 2012 (non-R2) as well
4. On my practical experience with (really) old versions of Freenas and Windows 2008 clustering, although persistent reservation tests were successfully passed, it still was not working in practice as soon as you try to move CSV from node to node.
5. I would say, this is not a normal prio. This simple problem is the only one issue stopping to use Freenas with Hyper-v clustering for real. So this is, kind of critical problem. I'm sure many people would try Freenas before purchasing TrueNas, since this is the same platform. And i'm sure they will never buy it then, faced to this issue in FreeNas. You should take it very serios, IMHO.

#27 Updated by Marco Müller almost 5 years ago

I stronly agree with Vladimirs arguments.

But it's not just Hyper-V. We considered to buy TrueNAS for SQL server cluster storage.
Furthermore each MS Failover Cluster that uses iSCSI is affacted. And there are many iSCSI usage scenarios, for example MS Exchange, Sharepoint, DHCP, DFS etc.
All these applications running on MS Failover Cluster with iSCSI disqualify FreeNAS from being used as enterprise storage, thus hold it stuck in private NAS boxes.

I observe this bug the whole 4 months now. Not providing a solution for this does not give a good shot about your support for enterprise needs.

I think this would probably not be the easiest thing to do, so maybe you should force the implementation of the new iSCSI stack.

BTW: I love FreeNAS anyway

#28 Updated by Steven Sedory almost 5 years ago

It's good to see that concern over this is growing.

I took your advice Nick and moved to Nexentastor for now. I just posted in a similar thread on the FreeNAS forum [[http://forums.freenas.org/index.php?threads/iscsi-3-persistent-reservations-under-hyper-v-server-2012.10738/#post-122007]] explaining this, as there are many more users represented there that are having to same frustrations.

I would be willing to give notable donations to the person or team who could fix this, if I knew doing so would get it moving along now. After this week though, I'm going to have to stick with Nexentastor and buy their Enterprise version (as production is not "allowed" on the community/16TB raw limit version).

This truly is, of course in my opinion (and many other's), thee only thing lacking in FreeNAS/TrueNAS as the perfect custom storage appliance software.

Can this please be escalated from "Priority: Normal" to something more critical?

#29 Updated by Jordan Hubbard almost 5 years ago

  • Target version changed from 72 to 9.2.1.6-RELEASE

Yeah. We think we may have a hail-mary play here that could rescue this. We're going to add a checkbox in the GUI to optionally enable CTL, which is already in the 9.2 branch and has been there for some time. If that then works with Windows 2012, huzzah. If it doesn't, then at least we can invest some time and energy into making CTL do the right thing rather than investing time into istgt, which is effectively a dead code branch as far as the FreeBSD project is concerned and something we'd really rather not mess with anymore. That "just works" for most values of (pre windows 2012) works and can represent a haven of stability for folks who don't want to jump onto the CTL bandwagon until it's been in production for a year or two.

#30 Updated by Steven Sedory almost 5 years ago

Hi Jordan, please excuse my ignorance.

What is CTL? Just a different daemon for iSCSI? If it is indeed a hail-mary, when would we expect to have that checkbox available?

Greatly appreciate all you do.

#31 Updated by Josh Paetzel almost 5 years ago

People aren't having the same (or any) issues with VMWare.

We've been working towards a new kernel iSCSI target for a couple years now. It's been in the backend of FreeNAS for about 6 months, we are just going to try surfacing it as an experimental feature, simply because we aren't going to sink any more developer time in to the current userland target. I've honestly no idea if the kernel target works with W2K12, but if it doesn't there's developer assets that are currently working on it and we can sort out whatever problems there may be.

In comparison, the current iSCSI target has a closed development model, and new versions aren't being exposed to the open source community anymore. (hint: There's a software project using a newer version of istgt than what is publicly available) Furthermore what little documentation there is is in Japanese. There's no bug tracker, mailing list, or way to get any attention from the sole developer short of emails that have been going unanswered for over a year now.

#32 Updated by Josh Paetzel over 4 years ago

  • Blocked by Feature #4929: Generate a config file for ctld added

#33 Updated by Josh Paetzel over 4 years ago

The kernel iSCSI target will be in the 9.2.1.6-BETA nightly tonight. Check the checkbox for enable experimental target in services -> iscsi, let the system reboot and see what happens.

Unlike istgt, the kernel target works best with zvols.

#34 Updated by Josh Paetzel over 4 years ago

  • Status changed from Screened to Investigation

#35 Updated by Steven Sedory over 4 years ago

Thank you for your work on this. I will setup a test environment when I can and give feedback.

#36 Updated by Jordan Hubbard over 4 years ago

BRB: Waiting for feedback from 9.2.1.6 nightlies.

#37 Updated by Chris Miller over 4 years ago

I am using FreeNAS in the latest VMware Workstation. 9.2.1.5 no issues other than not being able to validate cluster. My 2008 R2 cluster worked great except for moving CSV between nodes. After upgrading to 9.2.1.6BETA and enabling "Enable experimental target", I am no longer able to start the iSCSI service. I have tried both the 9.2.1.6 June 11th and the June 12th builds. I do have snapshots of each install point. All 3 builds have working iSCSI without the new option. It's only when I select that box in the 9.2.1.6BETA that the iSCSI service fails to start. I have rebooted.

#38 Updated by Josh Paetzel over 4 years ago

Can you run service ctld onestart from the CLI and attach the output of that?

#39 Updated by Chris Miller over 4 years ago

[root@freenas] /# service ctld onestart
Starting ctld.
ctld: error in configuration file at line 9 near '(': syntax error
ctld: configuration error; exiting
June 13 16:35:50 freenas ctld4343: configuration error; exiting
/etc/rc.d/ctld: WARNING: failed to start ctld

I had to type this out (no way to copy/paste) so hopefully no mistakes.

#40 Updated by Josh Paetzel over 4 years ago

Can you attach /etc/ctl to this ticket?

#41 Updated by Josh Paetzel over 4 years ago

Can you attach /etc/ctl.conf to this ticket?

#42 Updated by Chris Miller over 4 years ago

  • File ctl.conf added

Here ya' go!

#43 Updated by Josh Paetzel over 4 years ago

Interesting. Can you save your freenas config and mail it to me @ ? There may be sensitive stuff in the config so please don't attach it to the ticket. If you'd rather not do that I can send you the commands to extract the non-sensitive info I need from the database.

#44 Updated by Josh Paetzel over 4 years ago

It's almost as if the config generator script ran twice and appended to an existing config file, however that seems unlikely.

Can you:

rm /etc/ctl.conf
service ix-ctld start
service ctld start
And report any errors, plus attach /etc/ctl.conf if there are errors?

#45 Updated by Chris Miller over 4 years ago

Josh Paetzel wrote:

Interesting. Can you save your freenas config and mail it to me @ ? There may be sensitive stuff in the config so please don't attach it to the ticket. If you'd rather not do that I can send you the commands to extract the non-sensitive info I need from the database.

Where do I find the config file? Sorry, somewhat new to this products inner workings.

#46 Updated by Chris Miller over 4 years ago

  • File ctl.conf added

Josh Paetzel wrote:

It's almost as if the config generator script ran twice and appended to an existing config file, however that seems unlikely.

Can you:

rm /etc/ctl.conf
service ix-ctld start
service ctld start
And report any errors, plus attach /etc/ctl.conf if there are errors?

[root@freenas] /# cd /etc
[root@freenas] /etc# rm ctl.conf
[root@freenas] /etc# service ix-ctld start
[root@freenas] /etc# service ctld start
Starting ctld.
ctld: error in configuration file at line 9 near '(': syntax error
ctld: configuration error; exiting
June 13 17:54:46 freenas ctld3260: configuration error; exiting
/etc/rc.d/ctld: WARNING: failed to start ctld

#47 Updated by Josh Paetzel over 4 years ago

Just go to system settings and click the save config button.

#48 Updated by Josh Paetzel over 4 years ago

I've reopened 4929 as it appears the config file generator is stomping on itself.

#49 Updated by Josh Paetzel over 4 years ago

onerestart ignores whether it's enabled or not in rc.conf

Are you available for a webex tomorrow?

#50 Updated by Chris Miller over 4 years ago

I'm available now.

#51 Updated by Josh Paetzel over 4 years ago

Ok. How about in ~1 hr at 7am Pacific?

#52 Updated by Chris Miller over 4 years ago

That works for me.

#53 Updated by Josh Paetzel over 4 years ago

Chris,

I think we got it. Can you try out tonight's nightly?

#54 Updated by Chris Miller over 4 years ago

Sure thing. What time do the nightlies post?

#55 Updated by Chris Miller over 4 years ago

Josh. Just successfully validated a 2012 R2 Hyper-V cluster with the newest nightly.

THANK YOU!!!

#56 Updated by Josh Paetzel over 4 years ago

  • Status changed from Investigation to Resolved

Fantastic! So there's the word. You want w2k12 clustering use the experimental target.

Thanks everyone.

#57 Updated by James Knott over 4 years ago

It is still failing for me with 9.3 Alpha

#58 Updated by Steven Sedory over 4 years ago

Josh Paetzel wrote:

Fantastic! So there's the word. You want w2k12 clustering use the experimental target.

Thanks everyone.

Josh, can you please elaborate on this? I had to go to another Storage Appliance but would like to move back over if this is indeed stable. Is there an "experimental" in the iSCSI settings of FreeNAS now that is working with 2K12?

#59 Updated by Jordan Hubbard over 4 years ago

@James Knott - Are you saying that CTL (the "experimental target") is enabled on your system but still not working with 2K12?

#60 Updated by Josh Paetzel over 4 years ago

If you use a nightly image from 7/17 or later and enable the experimental target this will work now.

#61 Updated by Josh Paetzel over 4 years ago

  • Blocks Bug #5524: CTL should be default iSCSI in 9.3 added

#62 Updated by Steven Sedory over 4 years ago

Will run some tests soon. Thanks for your work on this!

#63 Updated by Steven Sedory over 4 years ago

Any idea when this will be included in an official build? I'd really like to move from another software to freenas for my datacenter.

#64 Updated by Jordan Hubbard over 4 years ago

The current provisional release date for 9.3 is October 15th. Sorry, but Kernel iSCSI is just one of many features planned for 9.3 and we don't have the engineering manpower to back-port all the necessary bits to the 9.2.1-BRANCH (or any current plans to release a 9.2.1.7). The quality of 9.3-ALPHA is currently very high as it has not diverged in any destabilizing ways from 9.2.1.6, however, so if I had to trust my data to 9.3-ALPHA, I would actually be more inclined to do that now than in, say, a month from now when a lot more changes have occurred.

#65 Updated by Steven Sedory over 4 years ago

Alright, thank you. I'll try today's build then. I appreciate the advice.

#66 Updated by Steven Sedory over 4 years ago

Jordan Hubbard wrote:

The current provisional release date for 9.3 is October 15th. Sorry, but Kernel iSCSI is just one of many features planned for 9.3 and we don't have the engineering manpower to back-port all the necessary bits to the 9.2.1-BRANCH (or any current plans to release a 9.2.1.7). The quality of 9.3-ALPHA is currently very high as it has not diverged in any destabilizing ways from 9.2.1.6, however, so if I had to trust my data to 9.3-ALPHA, I would actually be more inclined to do that now than in, say, a month from now when a lot more changes have occurred.

Can anyone confirm that the new iSCSI Kernal will be in 9.3? Is that still scheduled for October 15th?

#67 Updated by Josh Paetzel over 4 years ago

Confirmed.

It's also in 9.2.1.8 as the experimental target. (Checkbox in services -> iscsi -> global config)

#68 Updated by Steven Sedory over 4 years ago

Awesome. Thank you for all you do.

#69 Updated by Dru Lavigne 12 months ago

  • File deleted (Failover Cluster Validation Report-20140127.0753-filedata2.mht)

#70 Updated by Dru Lavigne 12 months ago

  • File deleted (dump.out)

#71 Updated by Dru Lavigne 12 months ago

  • File deleted (dump.out)

#72 Updated by Dru Lavigne 12 months ago

  • File deleted (ctl.conf)

#73 Updated by Dru Lavigne 12 months ago

  • File deleted (ctl.conf)

Also available in: Atom PDF