Project

General

Profile

Feature #24353

Indexing Service

Added by Ice BoosteR over 1 year ago. Updated 3 months ago.

Status:
Closed
Priority:
Nice to have
Assignee:
Andrew Walker
Category:
Services
Target version:
N/A
Estimated time:
24.00 h
Severity:
Low
Reason for Closing:
Not to be fixed
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Xeon E3-1225v3
24GB DDR3 ECC memory
4 WD red drives

Description

Hello guys,

I would really like to see an indexing service in FreeNAS, which is collecting the metadata of my stored files in a little database and I can use the data though the windows SMB share.

You could add a button in the "services" bar for that and give them the following options:
-database location
-datasets/pools which should be scanned
-list of file extensions or pattern which should be indexed
-time where the process starts and end with indexing (bachtjob)
-specify if the job should go to idle if there is user activity

Maybe this database could be stored to an USB Stick or SSD (new pool) or in the jails location...

Reason: I have lot of tagged photos and when I am searching for special photos with the tag "winter" the NAS need forever.

Related issues

Related to FreeNAS - Feature #25305: Add Diskover for disk-usage visualisation and indexed searchClosed
Is duplicate of FreeNAS - Feature #26619: Share file indexing serviceClosed: Duplicate2017-11-12
Has duplicate FreeNAS - Feature #18124: Spotlight indexing for macOS clients in SMBClosed

History

#1 Updated by John Hixson over 1 year ago

  • Status changed from Unscreened to Screened

#2 Updated by an odos over 1 year ago

Just an FYI, there are some patches for Samba that enable it to speak MS-WSP (windows search protocol). It requires sysutils/tracker like Samba's spotlight support. https://lists.samba.org/archive/samba-technical/2017-January/117990.html

The samba46 port installs with the "--with-spotlight" option.

root@TESTERER:/usr # smbd -b
<snip>
--with Options:
   WITH_ADS
   WITH_AUTOMOUNT
   WITH_DNS_UPDATES
   WITH_PAM
   WITH_PAM_MODULES
   WITH_PTHREADPOOL
   WITH_QUOTAS
   WITH_SENDFILE
   WITH_SPOTLIGHT
   WITH_SYSLOG
   WITH_WINBIND

So there's that. I haven't had time to look into the required config goo (i.e. the actual hard stuff). At some point in the future, this feature will enable spotlight for MacOS (that's some apple protocol tunneled over SMB, right?) and WSP support for windows clients. I believe the latter is very much a WIP, but the former would be great to have in FreeNAS. The topic of indexing files / metadata comes up fairly regularly in the forums. This would be a very nice feature to have.

#3 Updated by John Hixson over 1 year ago

I attempted this a year or two back. I was this close (for spotlight working). I stopped when the dependency list got really large and started pulling in things like X and cairo. I will revisit this again when I have some time hahahahaha. I will also investigate these samba patches that that speak MS-WSP. If such patches exist, I am curious why they are not in upstream as we should definitely work with the samba team to get them into a state where they will accept them.

#4 Updated by an odos over 1 year ago

John Hixson wrote:

I attempted this a year or two back. I was this close (for spotlight working). I stopped when the dependency list got really large and started pulling in things like X and cairo.

Well, I'll admit I omitted that "minor" detail in my comment. :-) As things stand right now, tracker has a pretty ridiculously large dependency list.

#5 Updated by an odos over 1 year ago

  • File pkg info output from jail.txt added

Attached "pkg info" output from jail on FreeNAS system after building sysutils/tracker and net/samba46 "--with-spotlight" from ports. It's pretty long.

#root@TESTERER:/usr/home/awalker # pkg info

#6 Updated by an odos over 1 year ago

  • File deleted (pkg info output from jail.txt)

#7 Updated by John Hixson over 1 year ago

  • Target version changed from 11.0-U1 to 49

Yeah, that is way to much bloat for this feature. I just can't do it ;-) I have no problem with an indexing service, but this just isn't realistic. When I do have some time I will further investigate other ways to do this, possibly even our own.

#8 Updated by an odos over 1 year ago

John Hixson wrote:

Yeah, that is way to much bloat for this feature. I just can't do it ;-) I have no problem with an indexing service, but this just isn't realistic. When I do have some time I will further investigate other ways to do this, possibly even our own.

Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.

#9 Updated by Ice BoosteR over 1 year ago

an odos wrote:

Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.

Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.

And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)

#10 Updated by an odos over 1 year ago

Ice BoosteR wrote:

an odos wrote:

Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.

Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.

And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)

Well, the true list of dependencies may be somewhat smaller. I basically held down the 'enter' key while installing samba and tracker. The indexing application needs to be able to read and index metadata/data from various media and document file-types, and provide thumbnails (in a desktop context). Tracker itself is supposedly fairly lightweight (it's designed to work on mobile devices), but it is also a part of the freedesktop project and for this reason and the aforementioned problem pulls in a lot of dependencies.

On the other hand, developing something to take its place (and provide spotlight / WSP support) seems like a fairly massive undertaking. The samba project primarily focuses on implementing SMB / AD on Unix. Once upon a time they also made a Samba web-frontend (SWAT), but later realized that this was a bad idea because it lay outside their field of expertise and was a distraction (and so SWAT was swatted). This is the same sort of situation with regard to indexing. They'll probably prefer to let other software projects handle it.

#11 Updated by John Hixson over 1 year ago

an odos wrote:

Ice BoosteR wrote:

an odos wrote:

Understood, by the time you install all the dependencies you'll have turned FreeNAS into "PCBSD-lite". There is definitely interest in Spotlight / WSP support, and a proper indexing service. Quite a few users do things client-side, which is rather inefficient.

Oh well, that quite a long list of dependencys. Did not see that coming. I would really like to see that feature, and it would defenetly beeing a huge benefit if this could be implemented. Maybe the Samba-team will manage to get a more lightwight version for this.

And yes, I have seen users who have mapped their network shares as internal drive, to let Windows do his indexing stuff, which is very inefficent.
Just think about the bunch of users in the enterprise area, thousands of word documents and excel sheets, and then searching the "ONE", which takes some time... ;)

Well, the true list of dependencies may be somewhat smaller. I basically held down the 'enter' key while installing samba and tracker. The indexing application needs to be able to read and index metadata/data from various media and document file-types, and provide thumbnails (in a desktop context). Tracker itself is supposedly fairly lightweight (it's designed to work on mobile devices), but it is also a part of the freedesktop project and for this reason and the aforementioned problem pulls in a lot of dependencies.

On the other hand, developing something to take its place (and provide spotlight / WSP support) seems like a fairly massive undertaking. The samba project primarily focuses on implementing SMB / AD on Unix. Once upon a time they also made a Samba web-frontend (SWAT), but later realized that this was a bad idea because it lay outside their field of expertise and was a distraction (and so SWAT was swatted). This is the same sort of situation with regard to indexing. They'll probably prefer to let other software projects handle it.

You do have a point here. When I originally did this work, I did not distinguish between run depends vs build depends. I should revisit this to see. If all this crap is just for building but not necessary to run, I won't have a problem adding it.

#12 Updated by John Hixson over 1 year ago

  • Target version changed from 49 to 11.2-BETA1

#13 Updated by Dru Lavigne over 1 year ago

#14 Updated by Dru Lavigne over 1 year ago

  • Status changed from Screened to Closed: Duplicate

#15 Updated by Dru Lavigne over 1 year ago

  • Related to Feature #25305: Add Diskover for disk-usage visualisation and indexed search added

#16 Updated by Dru Lavigne over 1 year ago

  • Status changed from Closed: Duplicate to Screened

#17 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Target version changed from 11.2-BETA1 to 11.3

#18 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Status changed from Screened to Not Started

#19 Updated by John Hixson 11 months ago

  • Assignee changed from John Hixson to Timur Bakeyev

#20 Avatar?id=13649&size=24x24 Updated by Ben Gadd 10 months ago

  • Target version changed from 11.3 to Backlog

#21 Avatar?id=13649&size=24x24 Updated by Ben Gadd 9 months ago

  • Severity set to New

#22 Updated by John Hixson 8 months ago

  • Assignee changed from Timur Bakeyev to John Hixson

#23 Updated by John Hixson 7 months ago

  • Category changed from OS to Services

#25 Updated by John Hixson 7 months ago

  • Severity changed from New to Low

#26 Updated by Dru Lavigne 4 months ago

  • Assignee changed from John Hixson to William Grzybowski

#27 Updated by William Grzybowski 4 months ago

  • Has duplicate Feature #18124: Spotlight indexing for macOS clients in SMB added

#28 Updated by William Grzybowski 4 months ago

  • Assignee changed from William Grzybowski to Andrew Walker

#30 Updated by Dru Lavigne 3 months ago

  • Status changed from Not Started to Closed
  • Target version changed from Backlog to N/A
  • Reason for Closing set to Not to be fixed

Ice: thanks for the feature request. We're closing the ticket for now as it will be another year or so before we have the resources to put this on the roadmap--this is a large feature which would require us writing a lot of code. In the meantime, if you hear that Samba will provide more hooks on their roadmap, please let us know!

Also available in: Atom PDF