Project

General

Profile

Feature #27364

Add option to middleware to enable fast-list for supported cloud sync backends

Added by Jacob Crisman about 1 year ago. Updated 8 days ago.

Status:
Done
Priority:
Nice to have
Assignee:
Vladimir Vinogradenko
Category:
Middleware
Target version:
Estimated time:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

Currently the backblaze b2 cloud sync task uses up Class C transactions at an alarming rate. In the past 2 days, nearly 1.5 million transactions have been consumed via the b2_list_file_names command. This translates to about ~$5 worth of transactions per backblaze's pricing structure, and can get pretty expensive rather quickly.

Please add an option in the task setup menu to configure the polling rate similar to what is available with synology's implementation as seen in this article.

https://help.backblaze.com/hc/en-us/articles/224378407-Why-am-I-Reaching-Class-C-Transaction-Caps-with-Synology-


Risk
The risk is low given that its an option that can be turned on/off in case it does not work well for some providers

Acceptance Criteria
Once this is done fast list attribute should be available for providers supporting it.

It will use fewer transactions (important if you pay for them)
It will use more memory. Rclone has to load the whole listing into memory.
It may be faster because it uses fewer transactions
It may be slower because it can’t be parallelized

A note should be done in documentation about it.


Related issues

Related to FreeNAS - Bug #37358: Add fast-list option to new UI for supported cloud sync backendsReady for Testing
Related to FreeNAS - Feature #37368: Tooltips for cloud sync optionsUnscreened

Associated revisions

Revision 7915066d (diff)
Added by Vladimir Vinogradenko 7 months ago

feat(cloud_sync): Add --fast-list option

Ticket: #27364

History

#1 Updated by Jacob Crisman about 1 year ago

  • File debug-san01-20171221161659.txz added

#2 Updated by Dru Lavigne about 1 year ago

  • Category changed from 2 to OS
  • Assignee changed from Release Council to Vladimir Vinogradenko
  • Target version set to 11.3

#3 Updated by Vladimir Vinogradenko about 1 year ago

  • Status changed from Unscreened to Screened

Currently Cloud Syncs occur every N minutes/hours, depending on how you've set it up. Thus, we can't change any polling rate as you suggest because there is no polling.

What we can do is enable https://rclone.org/docs/#fast-list option (b2 provider supports it) that is promised to reduce number of listing files transactions.

#4 Updated by Jacob Crisman about 1 year ago

Ahh, wasn't aware that this was using rclone as its method. Fast-list is definately the way to go here. I was already doing our backblaze backup with rclone prior to 11.1 via a jail. Is it possible to set up a cron job to run the backup task with this option enabled as a workaround until the feature gets added?

#5 Updated by Vladimir Vinogradenko about 1 year ago

Yes, this will work. You may do ps ax | grep rclone when backup job is running to get our base command-line options and also a temporary config file path to start with.

#6 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Status changed from Screened to Not Started

#7 Avatar?id=13649&size=24x24 Updated by Ben Gadd 10 months ago

  • Target version changed from 11.3 to Backlog

#8 Updated by Vladimir Vinogradenko 10 months ago

  • Severity set to Medium

#9 Updated by William Grzybowski 8 months ago

  • Category changed from OS to Middleware

#10 Updated by William Grzybowski 8 months ago

  • Subject changed from Add Backblaze polling rate option to Cloud Sync Tasks menu to Provide option to enable fast-list for supported backends
  • Description updated (diff)
  • Target version changed from Backlog to 11.3

#11 Updated by Vladimir Vinogradenko 7 months ago

  • Related to Bug #37358: Add fast-list option to new UI for supported cloud sync backends added

#12 Updated by Vladimir Vinogradenko 7 months ago

  • Status changed from Not Started to In Progress

#13 Updated by Vladimir Vinogradenko 7 months ago

  • Status changed from In Progress to Ready for Testing

#14 Updated by Dru Lavigne 7 months ago

  • Subject changed from Provide option to enable fast-list for supported backends to Provide option to enable fast-list for supported cloud sync backends
  • Needs Merging changed from Yes to No

#15 Updated by Dru Lavigne 7 months ago

  • File deleted (debug-san01-20171221161659.txz)

#16 Updated by Dru Lavigne 4 months ago

#17 Updated by Dru Lavigne 29 days ago

  • Target version changed from 11.3 to 11.3-BETA1

#18 Updated by Dru Lavigne 28 days ago

  • Subject changed from Provide option to enable fast-list for supported cloud sync backends to Add option to middleware to enable fast-list for supported cloud sync backends

#19 Updated by Dru Lavigne 8 days ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No

Also available in: Atom PDF