Feature #24353
Indexing Service
Xeon E3-1225v3
24GB DDR3 ECC memory
4 WD red drives
Description
Hello guys,
I would really like to see an indexing service in FreeNAS, which is collecting the metadata of my stored files in a little database and I can use the data though the windows SMB share.
You could add a button in the "services" bar for that and give them the following options:
-database location
-datasets/pools which should be scanned
-list of file extensions or pattern which should be indexed
-time where the process starts and end with indexing (bachtjob)
-specify if the job should go to idle if there is user activity
Maybe this database could be stored to an USB Stick or SSD (new pool) or in the jails location...
Reason: I have lot of tagged photos and when I am searching for special photos with the tag "winter" the NAS need forever.
Related issues
History
#1
Updated by John Hixson over 1 year ago
- Status changed from Unscreened to Screened
#2
Updated by an odos over 1 year ago
Just an FYI, there are some patches for Samba that enable it to speak MS-WSP (windows search protocol). It requires sysutils/tracker like Samba's spotlight support. https://lists.samba.org/archive/samba-technical/2017-January/117990.html
The samba46 port installs with the "--with-spotlight" option.
root@TESTERER:/usr # smbd -b <snip> --with Options: WITH_ADS WITH_AUTOMOUNT WITH_DNS_UPDATES WITH_PAM WITH_PAM_MODULES WITH_PTHREADPOOL WITH_QUOTAS WITH_SENDFILE WITH_SPOTLIGHT WITH_SYSLOG WITH_WINBIND
So there's that. I haven't had time to look into the required config goo (i.e. the actual hard stuff). At some point in the future, this feature will enable spotlight for MacOS (that's some apple protocol tunneled over SMB, right?) and WSP support for windows clients. I believe the latter is very much a WIP, but the former would be great to have in FreeNAS. The topic of indexing files / metadata comes up fairly regularly in the forums. This would be a very nice feature to have.
#3
Updated by John Hixson over 1 year ago
I attempted this a year or two back. I was this close (for spotlight working). I stopped when the dependency list got really large and started pulling in things like X and cairo. I will revisit this again when I have some time hahahahaha. I will also investigate these samba patches that that speak MS-WSP. If such patches exist, I am curious why they are not in upstream as we should definitely work with the samba team to get them into a state where they will accept them.
#4
Updated by an odos over 1 year ago
John Hixson wrote:
I attempted this a year or two back. I was this close (for spotlight working). I stopped when the dependency list got really large and started pulling in things like X and cairo.
Well, I'll admit I omitted that "minor" detail in my comment. :-) As things stand right now, tracker has a pretty ridiculously large dependency list.
#5
Updated by an odos over 1 year ago
- File pkg info output from jail.txt added
Attached "pkg info" output from jail on FreeNAS system after building sysutils/tracker and net/samba46 "--with-spotlight" from ports. It's pretty long.
#root@TESTERER:/usr/home/awalker # pkg info Show
#6
Updated by an odos over 1 year ago
- File deleted (
pkg info output from jail.txt)
#7
Updated by John Hixson over 1 year ago
- Target version changed from 11.0-U1 to 49
Yeah, that is way to much bloat for this feature. I just can't do it ;-) I have no problem with an indexing service, but this just isn't realistic. When I do have some time I will further investigate other ways to do this, possibly even our own.
#8
Updated by an odos over 1 year ago
John Hixson wrote:
Yeah, that is way to much bloat for this feature. I just can't do it ;-) I have no problem with an indexing service, but this just isn't realistic. When I do have some time I will further investigate other ways to do this, possibly even our own.
Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.
#9
Updated by Ice BoosteR over 1 year ago
an odos wrote:
Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.
Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.
And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)
#10
Updated by an odos over 1 year ago
Ice BoosteR wrote:
an odos wrote:
Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.
Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.
And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)
Well, the true list of dependencies may be somewhat smaller. I basically held down the 'enter' key while installing samba and tracker. The indexing application needs to be able to read and index metadata/data from various media and document file-types, and provide thumbnails (in a desktop context). Tracker itself is supposedly fairly lightweight (it's designed to work on mobile devices), but it is also a part of the freedesktop project and for this reason and the aforementioned problem pulls in a lot of dependencies.
On the other hand, developing something to take its place (and provide spotlight / WSP support) seems like a fairly massive undertaking. The samba project primarily focuses on implementing SMB / AD on Unix. Once upon a time they also made a Samba web-frontend (SWAT), but later realized that this was a bad idea because it lay outside their field of expertise and was a distraction (and so SWAT was swatted). This is the same sort of situation with regard to indexing. They'll probably prefer to let other software projects handle it.
#11
Updated by John Hixson over 1 year ago
an odos wrote:
Ice BoosteR wrote:
an odos wrote:
Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.
Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.
And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)Well, the true list of dependencies may be somewhat smaller. I basically held down the 'enter' key while installing samba and tracker. The indexing application needs to be able to read and index metadata/data from various media and document file-types, and provide thumbnails (in a desktop context). Tracker itself is supposedly fairly lightweight (it's designed to work on mobile devices), but it is also a part of the freedesktop project and for this reason and the aforementioned problem pulls in a lot of dependencies.
On the other hand, developing something to take its place (and provide spotlight / WSP support) seems like a fairly massive undertaking. The samba project primarily focuses on implementing SMB / AD on Unix. Once upon a time they also made a Samba web-frontend (SWAT), but later realized that this was a bad idea because it lay outside their field of expertise and was a distraction (and so SWAT was swatted). This is the same sort of situation with regard to indexing. They'll probably prefer to let other software projects handle it.
You do have a point here. When I originally did this work, I did not distinguish between run depends vs build depends. I should revisit this to see. If all this crap is just for building but not necessary to run, I won't have a problem adding it.
#12
Updated by John Hixson over 1 year ago
- Target version changed from 49 to 11.2-BETA1
#13
Updated by Dru Lavigne over 1 year ago
- Is duplicate of Feature #26619: Share file indexing service added
#14
Updated by Dru Lavigne over 1 year ago
- Status changed from Screened to Closed: Duplicate
#15
Updated by Dru Lavigne over 1 year ago
- Related to Feature #25305: Add Diskover for disk-usage visualisation and indexed search added
#16
Updated by Dru Lavigne over 1 year ago
- Status changed from Closed: Duplicate to Screened
#17
Updated by Kris Moore about 1 year ago
- Target version changed from 11.2-BETA1 to 11.3
#18
Updated by Kris Moore about 1 year ago
- Status changed from Screened to Not Started
#27
Updated by William Grzybowski 4 months ago
- Has duplicate Feature #18124: Spotlight indexing for macOS clients in SMB added
#30
Updated by Dru Lavigne 3 months ago
- Status changed from Not Started to Closed
- Target version changed from Backlog to N/A
- Reason for Closing set to Not to be fixed
Ice: thanks for the feature request. We're closing the ticket for now as it will be another year or so before we have the resources to put this on the roadmap--this is a large feature which would require us writing a lot of code. In the meantime, if you hear that Samba will provide more hooks on their roadmap, please let us know!