FreeNAS

thank you Sean for looking into it and for the insight that this dataset is supposed to be there. But still: the logfile gets spammed with that statvfs errors every 10s. It would be great if that could be suppressed somehow if the root of the pool is replicated. Should I open another ticket for that? Also I would like to remove my debug log and make this ticket public to link it in the forums, since there were a few complaints about this behavior where no one had a clue that this is "normal". Would that be okay?

How is this not a replication bug? Everything is fine until replication mess around with mountpoints with ZFS send/recv. What am I misunderstanding here?

Do you think it is okay to have a message nagging you every 10 seconds about a mountpoint that should not be there but isn't?

Why filesystems are mounted on the top of the other (wrong order)?

What makes you think it is a collectd bug if collectd is unaware of replication? It is simply trying to do its job, which is look at a mountpoint that should be there but isn't.

1. You are misunderstanding how replications work. It's counter-intuitive and annoying.
2. I don't, but the simplest and easiest way to fix that is to have collectd stop complaining about filesystems that it can't access.
3. They are not mounted in the wrong order; the root dataset is normally not mounted, but when you tell a zfs replication to mount the datasets it receives, then it mounts them -- and the root dataset is thus mounted. And then the subsequent datasets are mounted.
4. It's a collectd bug because there are lots of situations where you can have this happen, and there's not a lot of point to complaining about it. But I did not actually say (until now) that it's a collectd bug, I simply said that's what's complaining about it and is the easist, most doable thing to change.
5. Note that it only happens when you replicate an entire pool; it won't happen if you replicate a particular dataset (even recursively).
6. Alternately we could tell the replication to not mount the filesystems on the receive side. Then that would be another complaint people have, that they cannot access their data on the target system. This would mean using the '-u' option to the zfs recv command. Again, when I weigh the two options here, the one least likely to cause problems is still changing collectd to not complain about a filesystem it can't access.

1. I understand how replications work. I have worked with it for years.
2. collectd is doing its job, removing a feature of it is an ugly workaround.
3. The user can't access the mount point, either, right? Something does not look right.
4. Doesn't make it a bug. Its a real complaint, it is warning the user as it should. "Hey, I went to look for data in that path you told me to, but its not there!"
5. Doesn't make it less of a bug
6. Agreed, we do not want it to not mount on the receiving side.

Which brings back to: it is a replication issue. It might not be a bug in the replication code itself (autorepl.py), but it is part of the replication ecosystem, thus a bug in the replication.

You can choose to either mount none of the datasets, or all of them.

Not mounting them already results in complaints.

My position is not going to change on this.

Adding '-u' is causing unexpected (to me) behaviour. Will update after I've had some actual snapshots taken and replicated.

Doesn't -u do not mount the filesystem on receiving side? I thought we agreed that this wasn't the best thing to do. Or am I misunderstanding this again?

As I said: "Adding '-u' is causing unexpected (to me) behaviour."

You are the one insisting on a potentially dangerous resolution to a problem that would be most easily solved by not logging a pointless error. I don't like that I can't reconcile the documented behaviour of the flag with the observed results of the flag.

Sean Fagan wrote:

As I said: "Adding '-u' is causing unexpected (to me) behaviour."

You are the one insisting on a potentially dangerous resolution to a problem that would be most easily solved by not logging a pointless error. I don't like that I can't reconcile the documented behaviour of the flag with the observed results of the flag.

You said that, and after that you committed the code :).

I am not insisting on anything potentially dangerous. It is a real problem that needs real attention.
And yes, I do not agree with silencing a warning of something real. The data is not visible to the user on the receiving side, thats the real issue.

THERE IS NO DATA ON THE RECEIVING SIDE.

The root dataset exists solely to hold dataset properties (as opposed to pool properties). When you set a dataset property on ${POOL}, it's actually setting it on ${POOL}/${POOL}. Despite that, there is no data in it. It is an empty filesystem, even if you have data in the root of the pool.

Sean Fagan wrote:

THERE IS NO DATA ON THE RECEIVING SIDE.

The root dataset exists solely to hold dataset properties (as opposed to pool properties). When you set a dataset property on ${POOL}, it's actually setting it on ${POOL}/${POOL}. Despite that, there is no data in it. It is an empty filesystem, even if you have data in the root of the pool.

May  9 14:32:07 generator collectd[3367]: statvfs(/mnt/hotswap/Backup/SSDjails-backup/SSD-jails) failed: No such file or directory

There is no data in /mnt/hotswap/Backup/SSDjails-backup/SSD-jails ?

Correct.

collectd7892: statvfs(/mnt/ZFS_Backup_2/SSD_Pool_SS/jails/owncloud1) failed: No such file or directory

What about this? There is no data in /mnt/ZFS_Backup_2/SSD_Pool_SS/jails/owncloud1 as well?

If it's the same thing, then yes. But the logs are gone now so I can't tell what is going on there.

I'm waiting to see your patch to ZFS to handle this situation.

My patch? I thought you handled Replication and is a kernel developer. Why are you asking me for a patch?

Am I not allowed to point out problems?

On my systems:

Target:

root@mininas:/mnt # fgrep statvfs /var/log/messages | tail -2
Jul 14 12:18:54 mininas collectd[2947]: statvfs(/mnt/Storage/NAS/NAS) failed: No such file or directory
Jul 14 12:18:54 mininas collectd[2947]: statvfs(/mnt/Storage/NAS/Test2) failed: No such file or directory
root@mininas:/mnt # zfs list -r Storage/NAS | grep Test2
Storage/NAS/Test2                                        128K  3.63T   128K  /mnt/Storage/NAS/Test2

Source:

root@nas:/mnt/NAS # zfs list -r NAS | grep Test2
root@nas:/mnt/NAS # 

So:
a) The -u doesn't seem to have fixed it, so oh well, verifying that that is the case and then I'll revert it.
b) The dataset it's complaining about was deleted on the source side, but the replication did not delete it on the remote side. I believe that's because the current replicator seems to iterate over each dataset, instead of using 'zfs send -R'.

William Grzybowski wrote:

My patch? I thought you handled Replication and is a kernel developer. Why are you asking me for a patch?

Am I not allowed to point out problems?

When you're making them up and deciding that simple work-arounds for non-existent problems aren't acceptable, no.

Glad to know you think a bug reported by multiple people is non-existent.

Feel free to masquerade the error as you please. I am simply stating my opinion.

I wont interfere anymore. Bye.

Guys - First of all, this has gotten a bit out of control. Take 5 everybody.

Next, the issue here. Do we fix replication to mount/umount things so they match what was on the sending side? Or do we hide the warning from collectd. I'd say perhaps we close this, but put as a feature goal for the replication, rewrite the ability to look at what has the mount property enabled/disabled and have it make sure it is mounted/unmounted on the remote properly.

Just papering over the issue with collectd seems icky :/

The mounting is done automatically by zfs. You cannot change properties on a snapshot (which is what is replicated).

collectd is complaining about something that is not really an error: a filesystem listed in the kernel is not accessible. This could be due the filesystem being unmounted between the time it called getfsstat() and when it calls statvfs(), or it could be something has mounted over it, or it could be due to running in a jail or chrooted environment, or (not in *bsd with zfs, but could happen with nfs mounts) it could be due to permissions. Regardless, it's not a useful error message in a large percentage of the cases it would happen on.

1. The ability to mount on top of things is inherent in unix. By design and definition, things under it are hidden.
2. The kernel does not hide hidden mounts, nor does it adjust the mount table in a chroot.
3. This behaviour has always been there.
4. Replication can be done by using a recursive send, or iterating over the datasets; there are advantages to both methods (the iteration method can better handle the case where children datasets don't have the same set of snapshots that the parents do, but it means that deleted datasets aren't deleted). The current code uses iteration, which means that the only way a deleted dataset will end up being deleted on the target side is by "ssh ${user}@${host} zfs destroy -r ${remote_fs}/${dataset}", which frankly is a lot scarier than seeing log messages every 10 minutes.

I'm going to apologise to william and take back most of what I said above: I was looking at the wrong information when I concluded that ${POOL}/${POOL} was part of zfs. It's us, and goes back to 2016. The commit was 059424b57d7bb5cf02a01ae3dca7139d9e75141f, and I am trying to understand why Josh made it. (I've currently got it changed on my home system, and it seems to be working properly, but I've got to assume I'm missing something.)

I've created a new FIX-23871 branch, but not merged it to master.

The code works (as far as my testing has revealed so far), but it requires that the target be a dataset. That is, remotefs="Storage/NAS" localfs="NAS" works, but remotefs="Storage" localfs="NAS/SEF" will create Storage/SEF, instead of Storage/NAS/SEF. I think this is correct behaviour, but it may be that we need to be able to replicate to a pool. In which case I need to know, if I set localfs=NAS and remotefs=Storage, what should the result be? Storage/NAS, or just everything on Storage? What if I set localfs=NAS/SEF and remotefs=Storage -- should it still end up as Storage/NAS/SEF, or Storage/SEF?

Let me amend that: no, NAS/SEF -> Storage needs to end up as Storage/NAS/SEF, because of zfs. So the dataset needs to be created in that case. So the question is whether a pool name alone is allowed for the replication target.

I'm seeing similar messages on a new FreeNAS 11.0-U1 system that is a replication target for another system of the same OS level. Previous systems running FreeNAS-9.10.2-U1 & FreeNAS-9.10.2-U3 with a similar configuration (the main difference is the 9.10.2 systems are not doing recursive replication) do not report this.

Some addition details from poking around in /var/log/messages

- It repeats every 10 seconds
- It is currently logging 3 messages each interval - 1 for the top-level dataset (/mnt/tank/Replication_Target01/top-level-dataset) & 2 for child datasets (that also happen to be children of another child-dataset with the top-level dataset - so /mnt/tank/Replication_Target01/child-dataset1/child-dataset2 /mnt/Mirror01/Replication_Target01/child-dataset1/child-dataset3)
- There are several other child-datasets that do not hold child-datasets under them that the message is not logged for

Yes, the issue was diagnosed up above.

Ah, whoops I read most of the history but missed that last piece.

Dru Lavigne wrote:

John: please review this code or pass to someone else for review.

I was told I could take time on this one. It's targetted for 11.1. If you want to assign to someone else, feel free to do so.

I've reviewed the ticket history. I'll review the code tomorrow.

Let me summarize, again, what's going on, and why this is not a bug, but a design decision we chose a long time ago:

1. When replicating from source:tank/ReplSource to destination:tank/ReplTarget, the destination has to exist.
2. To ensure it exists, the replication code does a "zfs create -o readonly=on tank/ReplTarget" on the destination system.
3. It then replicates, using essentially "zfs send tank/ReplSource | zfs recv -dF tank/ReplTarget"
4. zfs then replicates the dataset ReplSource into the dataset tank/ReplTarget, which means that it is creating tank/ReplTarget/ReplSource
5. Because tank/ReplTarget was created by the replication code as read-only, it is not possible for zfs to create the mountpoint "tank/ReplTarget/ReplSource", and as result of this, the dataset tank/ReplTarget/ReplSource cannot be mounted in tank/ReplTarget. It is there however.
6. After rebooting (or exporting and re-importing) on the destination, zfs automatically creates tank/ReplTarget/ReplSource and mounts the dataset there.
7. There's some magic I'm not fully comprehending yet when the next replication runs that resets it back to the second previous case, making it look like the dataset has gone when it hasn't.

The solution is: create the target dataset when setting up replication.

Now, we can change the code to create the intervening datasets without readonly. But then this causes problems later on, because a modified dataset getting incremental snapshots won't work. (And I am hazy on remembering the details here, because we touched that code last when Josh Paetzel was here, and we came to the current code after a fair bit of discussion and testing.)

Sean Fagan wrote:

Let me summarize, again, what's going on, and why this is not a bug, but a design decision we chose a long time ago

• TrueNAS customer planned to export replication target via NFS
• TrueNAS customer created a replication task and fumbled the case of one of the path components to the target dataset (one path component was upper case instead of lower case).
• This caused the replication code to create the missing path components which resulted in the replicated dataset being unmountable.
• The above situation was not reported in the GUI and was not discovered until customer could not find dataset in TrueNAS UI. Fix was also not immediately apparent.

I think It would be good to have a GUI warning when the target path does not exist on the destination or some other form of safety belt.

They can fix the problem by making the intervening dataset(s) writable, by doing "zfs set readonly=off ${dataset}". (For the example I gave, on the destination system, "zfs set readonly=off tank/ReplTarget")

The GUI can try to warn about the path -- that's at a different level than autorepl.py, so that'd be a new feature to add. But the replication code just blatantly tries to create it, because that most easily handles the situation where the target to changed, destroyed, or otherwise has to have the intermediate components created. (And if the components already exist, it is a no-op -- it doesn't change the readonly attribute.)

Mostly I think this is documentation at this point.

I should also make this point: when just using replication for backing up (either locally or offsite), the fact that it's not automatically visible doesn't matter. You can even replicate when it's not mountable to another location, or back to the source.

When trying to use replication for visible redundancy (which is what I do at home, for example), then the additional step of creating the dataset, or of changing it to read-write, is necessary. I thought we'd agreed this needed to be documented better; does it still need to be called out more clearly? (Or in help in the UI?)

Sean Fagan wrote:

I thought we'd agreed this needed to be documented better; does it still need to be called out more clearly? (Or in help in the UI?)

Sean, thanks for taking the time to explain this succinctly. I agree with you regarding the autorepl.py behavior now that I understand what is going on. I will file a feature request for extra seatbelts and let conversation about this go on from there. :) FYI, I did resolve the issue on the customer system by changing the 'readonly' property of the path to the replication target and then mounting the datasets.

The current docs state the following:

The target dataset on the receiving system is automatically created in read-only mode to protect the data. To mount or browse the data on the receiving system, create a clone of the snapshot and use the clone. Clones are created in read/write mode, making it possible to browse or mount them. See Snapshots for more information on creating clones.

I think part of the reason the customer was confused was because some of his replicated datasets were visible and others were not visible. The current docs do not address this aspect of replication behavior (specifically, whether it will be visible / browsable depends on the ZFS readonly property of the path to the dataset). Perhaps they need to be a bit more nuanced. It might be a good idea to add to the tooltip as well.

We can add to the existing note, but people continue to encounter this, which means it is not expected. I suspect we need to either add an option to the GUI, or have the replication code set the ZFS property, or maybe both.