Project

General

Profile

Feature #21333

Provide some way to expose all snapshots via shadow-copy2, not just a subset of snapshots

Added by Stilez y over 3 years ago. Updated over 1 year ago.

Status:
Done
Priority:
Nice to have
Assignee:
Andrew Walker
Category:
Services
Estimated time:
Severity:
Low Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

I'd like to create a snapshot routine that looks something like this, and export it via shadow-copy ("Previous Versions") on CIFS:

  • Every 30 minutes, kept for 72 hours
  • Every 12 hours, kept for 21 days
  • Weekly, kept for 6 months
  • Offline or further snapshots for 1-3 years or as needed, or backups in addition (this list is only as an example)

Creating such a routine in 9.10.2 is easy in principle but there's two problems. First, the GUI doesn't allow much flexibility in snapshot interval (e.g., 1 day/1 week/1 month). Second, if multiple tasks are needed for a snapshot routine (as here), the GUI only seems to allow 1 of them at most to be selected and exported under "Previous Versions" via CIFS, so the client can't see all snapshots available, and exposing newer ones means not exposing older ones.

Google reveals a comment on Github (https://github.com/zfsonlinux/zfs-auto-snapshot/issues/10#issuecomment-46753124) that the latter issue can be worked around by clever choice of snapshot naming, although it doesn't look like it can be done at present in FreeNAS, or might mess around with FreeNAS expectations on snapshot naming, causing other issues:

One can expose all weekly, monthly, hourly snapshots into windows if you're clever with the snapshot naming.. use a number system such as 01, 02, 03 ,04 for week/mon/hour/day etc.. then use the format property to interpret that label as part of the date-time

vfs objects = shadow_copy2
shadow: snapdir = .zfs/snapshot
shadow: sort = desc
shadow: format = zfs-auto-snap_%S-%Y-%m-%d-%H%M
shadow: localtime = yes

too easy :)

Another possible route might be to add a checkbox that activates the vfs-shadow-copy2 setting "shadow:snapdirseverywhere = yes" to automagically solve everything, although I'm not 100% sure if this will solve the issue or have any side effects. But either way it seems useful to fix this, and unhelpful that only a subset of snapshots can be selected to be exposed to the SMB client.

Summary - It seems technically possible, and an important/useful ability, to expose all snapshots for a share via Previous Versions, not just one subset of snapshots, but not doable within the GUI. Can a way to do this be exposed in the GUI?

sshot-1.png (37.1 KB) sshot-1.png ahej ahej, 04/30/2017 07:28 AM
sshot-3.png (37.1 KB) sshot-3.png ahej ahej, 04/30/2017 07:28 AM
sshot-4.png (36.9 KB) sshot-4.png ahej ahej, 04/30/2017 07:28 AM
sshot-2.png (37 KB) sshot-2.png ahej ahej, 04/30/2017 07:28 AM
10925
10926
10927
10928

Related issues

Copied to FreeNAS - Feature #75873: Add support for nested ZFS datasets via shadow_copy_zfs vfs moduleDone

History

#1 Updated by Bonnie Follweiler over 3 years ago

  • Assignee set to William Grzybowski

#2 Updated by William Grzybowski over 3 years ago

  • Status changed from Unscreened to Screened
  • Priority changed from No priority to Nice to have
  • Target version set to 9.10.4

#3 Updated by an odos over 3 years ago

In principle, can't this be solved by using the new shadow_copy2 parameter "shadow:snapprefix", which was introduced in Samba 4.5.0?
https://bugs.freenas.org/issues/19298

#4 Updated by Stilez y over 3 years ago

In principle it looks like it could, but I'm cautious because TBH, I don't fully understand the new smb4.conf options and the kinds of ways they can be used. (help info is scant and examples absent). Also not sure to what extent snapdirseverywhere is relevant as that also seems to allow some kind of trawling of multiple snapshot sets/dirs?

Bug pages aren't the best place to ask for help so I'll go educate myself; my main thought is if they are useful (and especially if snapprefix is more powerful than it seems!) can we expose this capability usefully and explicitly in the UI with some kind of more open option: "Which snapshots to expose to user?", allowing the user some kind of logic to choose what is shown, rather than just choosing one of a list of snapshot tasks. Not sure what logic would help as I don't yet fully understand the .conf options and how they can be used. Can we also allow an option to customise the snapshot names format for a snapshot task, so that shadow_copy2 options that are based on snapshot name, can be used more flexibly.

#5 Updated by Per Nils Snygg over 3 years ago

My solution was to consolidate the names of the snapshots routines into a single namespace.

https://forums.freenas.org/index.php?threads/collect-multiple-periodic-snapshot-for-restore-previous-version-under-cifs.51129/

It works and I'm using this in live production.

#6 Updated by William Grzybowski over 3 years ago

  • Assignee changed from William Grzybowski to John Hixson

#7 Avatar?id=14398&size=24x24 Updated by Kris Moore over 3 years ago

  • Target version changed from 9.10.4 to 11.1

#8 Updated by ahej ahej over 3 years ago

10925
10926
10927
10928

This Feature is very usefull,and We needed it.
这个功能很有用,而且我们需要它。

I think this canbe solved by use the method of FreeNAS corral.
我认为可以用FreeNAS corral的方法来解决这个问题。

I tested in FreeNAS-Corral-10.0.5,
我用FreeNAS-Corral-10.0.5进行过测试,

    1.create myz/docs
    2.create share docs
    3.create snapshots
        every 1 min,kept 10min
        every 5 min,kept 1hour
        every 15min,kept 6hour
        every 1hour,kept 1day

freenas-corral cli:

unix::/calendar/snapshot>show
 Name           Schedule       Enabled        Dataset        Lifetime       Recursive      Prefix         Replicable

 e1min10min     second:0       yes            myz/docs       600            no             e1min10min-aut yes
                                                                                           o
 e5min1hour     minute:*/5,    yes            myz/docs       3600           no             e5min1hour-aut yes
                second:0                                                                   o
 e15min6hour    minute:*/15,   yes            myz/docs       21600          no             e15min6hour-au yes
                second:0                                                                   to
 e1hour1day     minute:48,     yes            myz/docs       86400          no             e1hour1day-aut yes
                second:0                                                                   o
unix::/calendar/snapshot>e1min10min show
 Property    Description        Value        Settable
name         Name          e1min10min        yes
schedule     Schedule      {"second": "0"}   yes
enabled      Enabled       yes               yes
dataset      Dataset       myz/docs          yes
lifetime     Lifetime      600               yes
recursive    Recursive     no                yes
prefix       Prefix        e1min10min-auto   yes
replicable   Replicable    yes               yes
unix::/calendar/snapshot>e5min1hour show
 Property    Description                Value                 Settable
name         Name          e5min1hour                         yes
schedule     Schedule      {"minute": "*/5", "second": "0"}   yes
enabled      Enabled       yes                                yes
dataset      Dataset       myz/docs                           yes
lifetime     Lifetime      3600                               yes
recursive    Recursive     no                                 yes
prefix       Prefix        e5min1hour-auto                    yes
replicable   Replicable    yes                                yes
unix::/calendar/snapshot>e15min6hour show
 Property    Description                 Value                 Settable
name         Name          e15min6hour                         yes
schedule     Schedule      {"minute": "*/15", "second": "0"}   yes
enabled      Enabled       yes                                 yes
dataset      Dataset       myz/docs                            yes
lifetime     Lifetime      21600                               yes
recursive    Recursive     no                                  yes
prefix       Prefix        e15min6hour-auto                    yes
replicable   Replicable    yes                                 yes
unix::/calendar/snapshot>e1hour1day show
 Property    Description                Value                Settable
name         Name          e1hour1day                        yes
schedule     Schedule      {"minute": "48", "second": "0"}   yes
enabled      Enabled       yes                               yes
dataset      Dataset       myz/docs                          yes
lifetime     Lifetime      86400                             yes
recursive    Recursive     no                                yes
prefix       Prefix        e1hour1day-auto                   yes
replicable   Replicable    yes                               yes

freenas-corral shell:

[root@freenas] /mnt/myz/docs/.zfs/snapshot# ls
./                              e1min10min-auto-20170430.1405/  e5min1hour-auto-20170430.1325/
../                             e1min10min-auto-20170430.1406/  e5min1hour-auto-20170430.1330/
e15min6hour/                    e1min10min-auto-20170430.1407/  e5min1hour-auto-20170430.1335/
e15min6hour-auto-20170430.1300/ e1min10min-auto-20170430.1408/  e5min1hour-auto-20170430.1340/
e15min6hour-auto-20170430.1315/ e1min10min-auto-20170430.1409/  e5min1hour-auto-20170430.1345/
e15min6hour-auto-20170430.1330/ e1min10min-auto-20170430.1410/  e5min1hour-auto-20170430.1350/
e15min6hour-auto-20170430.1345/ e1min10min-auto-20170430.1411/  e5min1hour-auto-20170430.1355/
e15min6hour-auto-20170430.1400/ e1min10min-auto-20170430.1412/  e5min1hour-auto-20170430.1400/
e15min6hour-auto-20170430.1415/ e1min10min-auto-20170430.1413/  e5min1hour-auto-20170430.1405/
e1hour1day/                     e1min10min-auto-20170430.1414/  e5min1hour-auto-20170430.1410/
e1hour1day-auto-20170430.1348/  e1min10min-auto-20170430.1415/  e5min1hour-auto-20170430.1415/
e1min10min-auto-20170430.1403/  e5min1hour-auto-20170430.1315/
e1min10min-auto-20170430.1404/  e5min1hour-auto-20170430.1320/

OK,it looks worked well.See sshot-1.png,sshot-2.png,sshot-3.png,sshot-4.png.
好了,现在看起来工作正常。看sshot-1.png,sshot-2.png,sshot-3.png,sshot-4.png这几张截图。

Because FreeNAS-Corral and FreeNAS 9.10 difference is very big,Integration may take some time.
因为FreeNAS-Corral和FreeNAS 9.10的方法差别很大,整合可能会需要一些时间。

Thanks for FreeNAS team.
感谢FreeNAS开发小组。

#9 Updated by Stilez y about 3 years ago

One trivially easy solution to half of this, would be to create an option in the "periodic snapshots" form, to choose one of 3 formats for the snapshot name:

  1. Snapshot has the current name and date format (for backwards compatibility).
  2. Snapshot is named using UTC time (equivalent to <SNAPSHOT_TITLE>_`date -uj +"%Z-%Y.%m.%d-%H.%M.%S`)
  3. Snapshot is named using local time (equivalent to <SNAPSHOT_TITLE>_`date -uj +"%Z-%Y.%m.%d-%H.%M.%S`). This has identical syntax but uses local timezone not UTC

The effect of the 2nd and 3rd options are that Samba shadow_copy2 would immediately be able to recognise and provide all snapshots that exist as File History, from all periodic snapshot jobs, and not just the snapshots from one periodic snapshot job.

The reason is that if the user chooses the second or third options, all snapshots of whichever kind would share the same date format. Samba shadow_copy2 can use such names with File History regardless of which periodic job created the snapshot, because it can natively split out a valid strftime part from a snapshot name, provided it occurs at the end of the snapshot name and in a consistent strftime-based format. But the current naming scheme has the timestamp at the start of the snapshot name, not the end, which shadow_copy2 can't handle.

EFFECT OF CHANGE:

Suppose the user has two periodic snapshots - hourly for 24 hours then daily for 4 weeks.

EXISTING SNAPSHOT NAMES:

  • auto-20171009.0800-01h
  • auto-20171009.0800-01d

Samba can't pass both of these to File History, so the user can only be shown snapshot versions from at most one of them (either the snapshots dated < 24 hours or those dated >= 24 hours to 4 weeks, not both)

PROPOSED EFFECT:

The proposed option would allow snapshots to be named alternatively as follows:

  • auto-01h_UTC-2017.10.09-08.00.00
  • auto-01d_UTC-2017.10.09-08.00.00

or for the local timezone option:

  • auto-01h_BST-2017.10.09-07.00.00
  • auto-01d_BST-2017.10.09-07.00.00

Samba's shadow_copy2 can be told to use '_' as a separator and the latter part as a strftime formatted string. Since the automatic snapshot jobs for all snapshots in both of these formats share the same format for the timestamp part of their name, shadow_copy2 can present all snapshots which use UTC or local time, to the user, from both periodic jobs, not just a partial selection. At the moment the user can at least set these as manual parameter if the snapshots could be named appropriately.

SUMMARY OF REQUESTED CHANGE

This would be a fairly trivial change - one UI dropdown option with 3 choices, added to the "periodic snapshots" form, probably one line of code that checks it and uses a different snapshot name based on the user selection (when snapshots are created), and a quick check for anything else relying on the snapshot name format to ensure it will recognise the new formats (perhaps only the timed destroy function?).

It wouldn't fix the time-flexibility issue, but it would at least let the server expose all snapshots to the user in File History (not just one subset of them) which is arguably much more important.

Is it possible to slip ithis part of the 'fix' in early, into the next update (11.0-U5 or 11.1) - even if the flexible periods and any related shadow_copy2 options aren't done until later - so that it can be used by those who would find it helpful?

#10 Avatar?id=14398&size=24x24 Updated by Kris Moore about 3 years ago

  • Target version changed from 11.1 to 11.2-BETA1

#11 Avatar?id=14398&size=24x24 Updated by Kris Moore almost 3 years ago

  • Target version changed from 11.2-BETA1 to 11.3

#12 Avatar?id=14398&size=24x24 Updated by Kris Moore over 2 years ago

  • Status changed from Screened to Not Started

#13 Updated by John Hixson over 2 years ago

  • Assignee changed from John Hixson to Timur Bakeyev

#14 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 2 years ago

  • Target version changed from 11.3 to Backlog

#15 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 2 years ago

  • Severity set to New

#16 Updated by Timur Bakeyev over 2 years ago

  • Severity changed from New to Low Medium

#18 Updated by Timur Bakeyev about 2 years ago

Few notes on this - there is another fix coming in upstream regarding the way how snapshot names are internally passed - that may affect the format to use.

Another thing is that it could be easier to teach shadow_copy2 to recognize multiple templates and interpret them accordingly.

The third thing to keep in mind is that due to the 88 bytes limitation on the mount path we'd try to keep snapshot name as minimalistic as possible, i.e. auto- is just a waste of 5 useful bytes, could be just A for example.

#19 Updated by Stilez y about 2 years ago

Timur Bakeyev wrote:

The third thing to keep in mind is that due to the 88 bytes limitation on the mount path we'd try to keep snapshot name as minimalistic as possible, i.e. auto- is just a waste of 5 useful bytes, could be just A for example.

If we're discussing upstream, then worth noting that the 88 byte limit will cease to exist AFAIK within 3 months or so in FreeBSD 12 (scheduled for release 2-6 Nov 2018), where ino64 brings 64 bit inodes and 1024 byte mountpoint struct limits in statfs.

While it'll take a little extra time for FreeNAS to use it, it seems the 88 byte limit shouldn't be a consideration for very much longer:

From rev 318736:

Extend the ino_t, dev_t, nlink_t types to 64-bit ints.  Modify
struct dirent layout to add d_off, increase the size of d_fileno
to 64-bits, increase the size of d_namlen to 16-bits, and change
the required alignment.  Increase struct statfs f_mntfromname[] and
f_mntonname[] array length MNAMELEN to 1024.

Relevant:
https://www.phoronix.com/scan.php?page=news_item&px=FreeBSD-64-bit-Inodes-ino64
https://svnweb.freebsd.org/base?view=revision&revision=318736

#20 Updated by Timur Bakeyev about 2 years ago

  • Category changed from OS to Services

#21 Updated by Dru Lavigne about 2 years ago

  • Assignee changed from Timur Bakeyev to John Hixson

#22 Updated by Dru Lavigne about 2 years ago

  • Assignee changed from John Hixson to William Grzybowski

#23 Updated by William Grzybowski about 2 years ago

  • Status changed from Not Started to Unscreened
  • Assignee changed from William Grzybowski to Andrew Walker

Andrew, this is something we should consider to bring it in and upstream:

https://github.com/freenas/corral-samba/blob/corral/master/source3/modules/vfs_shadow_copy_zfs.c

#24 Updated by ahej ahej almost 2 years ago

看到这个功能还在更新,我很高兴,因为等待这个功能已经5年了。最早知道这个功能的是napp-it,因为这个功能是包含在sun solaris系统内部的,不能拆分,所以也不能够移植到freenas。后来发现synology也有这个功能,他是修改了samba,所有这个也不能移植到freenas。后来发现这个“shadow_copy support in samba is limited to single snapshot formats”https://redmine.ixsystems.com/issues/1235跟此功能很像,但是这个最后还是关闭了。

这个特征目前有2大好处:
1、减少快照的数量,合理安排快照频率
据我自己从多个nas系统测试的结果,windows xp/7/8/10所能显示的快照数量在1200-1300之间,我现在使用的是15分钟一次快照,从08:30--18:30,周期一个月,也就是1200个快照,刚好所有的windows客户端都能正常显示。但是这里的快照很密集,而且是等频率的,显然时间越久,需要的快照频率越低,如果按照5分钟一次,持续2小时,1小时一次,持续2天,1天一次,持续2月,1周一次,持续6个月,在这样的快照模式下,一共有157个快照,跨度长达6个月。
2、从用户的角度看,简单是最好的老师
不需要太多的教程,教会用户如何使用快照。当管理员需要把一个重要的时间点做成一个快照,比如manual-20181108,这个是一个静止的快照,即使1年之后,自动快照已经把auto-20181108-2200删除了,但是manual-20181108还是在的,用户可以从“以前的版本”中看到这个manual-20181108,重要的文件信息都在。当然如果这个功能实现的话,快照名称就不会有限制,比如可以用veryimportant-update-20181108这样的名称命名,用户一看就知道含义,管理员只需要告诉用户找20181108这个日期的以前的版本,其他的就不用管了。

Translate the following:
I'm glad to see that this feature is still being updated, as it's been waiting for five years. It was first known as napp-it, and since the feature is contained within the sun solaris system, it cannot be split up, so it cannot be ported to freenas. It turns out that synology also has this feature, he's modified samba, and all of this can't be ported to freenas either. Later found that the "shadow_copy support in samba is limited to single snapshot formats" https://redmine.ixsystems.com/issues/1235 like this functionality, but the final is still closed.

This feature currently has two major benefits:
1. Reduce the number of snapshots and arrange snapshot frequency reasonably
According to my own results from multiple nas tests, the number of snapshots Windows xp/7/8/10 can display is between 1200 and 1300. I now use a snapshot every 15 minutes, from 08:30 to 18:30, with a period of 1 month, namely 1200 snapshots. Just all Windows clients can display normally. But the snapshot is very dense, and the frequency of such as are, apparently, the longer it is need a snapshot of the lower frequency, if according to the 5 minutes at a time, the last 2 hours, 1 hour time, last 2 days, 1 days, last 2 months, 1 time on Monday, for six months, in such a snapshot of the model, a total of 157 snapshot, span up to six months.
2. From the perspective of users, simplicity is the best teacher
It doesn't take too many tutorials to teach users how to use snapshots. When the administrator needs to make a snapshot of an important point in time, such as manual-20181108, this is a static snapshot. Even after one year, the automatic snapshot has deleted auto-20181108-2200, but manual-20181108 is still there. Users can see this manual-20181108 from the "previous version". Of course, if this function is implemented, there will be no limit to the snapshot name. For example, it can be named with such names as veryimportant-update-20181108. Users can see the meaning immediately.

另:不得不说,现在的机器翻译较前两年好多了,基本能翻译清楚我要表达的意思。
Ps: I have to say that the machine translation is much better than the previous two years. It can basically translate clearly what I want to express.
有了这个功能,freenas就是无敌的。
With this feature, freenas is invincible.

#25 Updated by ahej ahej almost 2 years ago

William Grzybowski wrote:

Andrew, this is something we should consider to bring it in and upstream:

https://github.com/freenas/corral-samba/blob/corral/master/source3/modules/vfs_shadow_copy_zfs.c

从技术层面上说,如果FreeNAS-Corral的方法可行,直接修改samba,就可以不修改freenas的中间件和webui,可以说是最省力的方法。你创建一个zfs快照,samba就给你显示出来,就这么简单,想怎么创建都可以,只要不超过1300个快照的极限。
On a technical level, if the freenas-corral approach works, modify samba directly, without modifying FreeNAS 'middleware and webui, arguably the most labor-saving approach. You create a ZFS snapshot, which samba shows you, as simple as that, as long as you don't exceed the limit of 1,300 snapshots.

#26 Updated by Andrew Walker almost 2 years ago

Thanks for bringing this up. I'll test today if I have time. I was actually looking into updating / testing the vfs_module that the corral one was based on. It's nice to not duplicate effort. :)

#28 Updated by Andrew Walker almost 2 years ago

  • Status changed from Unscreened to In Progress
  • Target version changed from Backlog to 11.3

This is a quick update on this once regarding the corral VFS module. I updated / fixed it to work with Samba 4.7 and 4.9. It seems to be working correctly, but needs some polish before it's put into FreeNAS. 11.3 is a reasonable target for it and I might be able to sneak it into an 11.2 U release.

#29 Updated by ahej ahej almost 2 years ago

Andrew Walker wrote:

This is a quick update on this once regarding the corral VFS module. I updated / fixed it to work with Samba 4.7 and 4.9. It seems to be working correctly, but needs some polish before it's put into FreeNAS. 11.3 is a reasonable target for it and I might be able to sneak it into an 11.2 U release.

Wonderful!

#30 Updated by Mauricio Silveira over 1 year ago

ahej ahej wrote:

Andrew Walker wrote:

This is a quick update on this once regarding the corral VFS module. I updated / fixed it to work with Samba 4.7 and 4.9. It seems to be working correctly, but needs some polish before it's put into FreeNAS. 11.3 is a reasonable target for it and I might be able to sneak it into an 11.2 U release.

Wonderful!

This one was INDEED a missing feature.
I could see from the the 11.2 releases changelogs that it wasn't "sneaked" into 11.2 .
Any news on this?

#31 Updated by Andrew Walker over 1 year ago

This is a quick update on this ticket. I spent a decent amount of time last week and over the weekend putting polish on the vfs module. Samba changed significantly since Corral and so I had to update the module accordingly. The corral module was also limited to a single ZFS dataset per share. The updated version now supports nested datasets. I was also unhappy with the sheer number of libzfs-related calls. Very inefficient. I introduced a configurable parameter to cache the mappings of <@GMT-token>/file -> realpath (defaults to on), so that they will only get called once when browsing a snapdir. I'll do some more testing and hopefully have it in the nightlies this week.

#32 Updated by Bug Clerk over 1 year ago

  • Status changed from In Progress to Ready for Testing

#33 Updated by Bug Clerk over 1 year ago

  • Target version changed from 11.3 to 11.3-BETA1

#34 Updated by Bug Clerk over 1 year ago

  • Copied to Feature #75873: Add support for nested ZFS datasets via shadow_copy_zfs vfs module added

#35 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Target version changed from 11.3-BETA1 to Master - FreeNAS Nightlies
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

Also available in: Atom PDF