Project

General

Profile

Feature #36303

officially supported kubernetes provisioner

Added by Travis Hansen almost 2 years ago. Updated almost 1 year ago.

Status:
Closed
Priority:
No priority
Assignee:
William Grzybowski
Category:
Services
Target version:
Estimated time:
Severity:
Low Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

Kubernetes supports stateful applications through what are referred to as PersistentVolumes (PV). Users can request storage through a PersistentVolumeClaim (PVC). PVCs can only be 'bound' or attached to pre-defined PVs unless the cluster operates a (or many) 'provisioners'. When running in cloud environments provisioners are generally integrated with the cloud providers' services (ie: in aws kubernetes detects a PVC and creates an appropriate block volume, then returns the newly created storage to kubernetes as a PV, which is subsequently bound to the PVC). Those running kubernetes and wanting to integrate it with FreeNAS currently would be required to manually create the actual volumes in FreeNAS, then subsequently manually create the associated PVs in kubernetes. It's cumbersome and doesn't scale. An officially supported solution would be fantastic.

While nfs could be helpful (a basic generic nfs provisioner already exists for this [2]), the most obvious integration would be iscsi. Each provisioner instance would likely be started knowing which pool (pre-created) along with some other details. Once the provisioner detects it has a new volume to supply I see the general workflow being something like the below (all through the http api):

1. create new zvol
2. create target
3. create target group
4. create extent
5. attach extent to target group

After all of that has been done, the provisioner returns in essence the iscsi connection/path details back to kubernetes (the rest is handled by kubernetes itself...formatting, mounting, etc, etc). Some of the items could be implied I suppose (similar to the pool as mentioned above). Anyway, that's the general idea.

Thanks for the consideration!

Some links describing the process and giving some ideas:
1. https://github.com/nmaupu/freenas-provisioner
2. https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client
3. https://github.com/kubernetes-incubator/external-storage/tree/master/iscsi/targetd
4. https://kubernetes.io/docs/concepts/storage/storage-classes/
5. https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/
6. https://github.com/kubernetes-incubator/external-storage/tree/master/docs/demo/hostpath-provisioner
7. https://netapp.io/2016/12/23/introducing-trident-dynamic-persistent-volume-provisioner-kubernetes/
8. https://blog.purestorage.com/pure-service-orchestrator-now-ga/


Related issues

Related to FreeNAS - Bug #34056: Unmount datasets before deletionDone
Related to FreeNAS - Feature #53039: Allow force deletion of datasets through v1 and v2 APIClosed

History

#2 Updated by Travis Hansen over 1 year ago

I actually spent some time and nearly rewrote the provisioner available here: https://github.com/nmaupu/freenas-provisioner I additionally wrote an iscsi variant available here: https://github.com/travisghansen/freenas-iscsi-provisioner (not as bullet proof currently, way more moving parts). The nfs variant is what I would consider close to production-ready (I'll be running in my own production setup). Long-term, we probably need to write completely new variants that work with csi spec but much of the work already done can be leveraged for those.

#3 Updated by Dru Lavigne over 1 year ago

  • Category changed from Middleware to Services
  • Assignee changed from Release Council to John Hixson
  • Target version changed from Backlog to 12.0

#5 Updated by John Hixson over 1 year ago

  • Severity changed from New to Low Medium

#6 Updated by John Hixson over 1 year ago

  • Status changed from Unscreened to Screened

#7 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from John Hixson to William Grzybowski

#8 Updated by Travis Hansen over 1 year ago

Running the nfs code in a semi-production state now for a few months. I've continued to find minor bug fixes etc. It's currently in pretty good shape and I'll be enabling in our CI environment shortly which will stress it out a fairly extensively (lots of churn).

I've not followed the new FreeNAS API much so there's currently lots of change going on for the long term vision of this idea (csi vs not, new FreeNAS API vs current).

Thanks for the consideration!

#9 Updated by Travis Hansen over 1 year ago

Don't see a related issues feature. Putting these here for reference.

https://redmine.ixsystems.com/issues/34056
https://redmine.ixsystems.com/issues/53039

#10 Updated by Dru Lavigne over 1 year ago

  • Related to Bug #34056: Unmount datasets before deletion added

#11 Updated by Dru Lavigne over 1 year ago

  • Related to Feature #53039: Allow force deletion of datasets through v1 and v2 API added

#12 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 1 year ago

  • Status changed from Screened to Closed

Also available in: Atom PDF