Project

General

Profile

Feature #27364

Add fast-list support to legacy UI for supported cloud sync backends

Added by Jacob Crisman over 1 year ago. Updated 24 days ago.

Status:
Ready for Testing
Priority:
Nice to have
Assignee:
Vladimir Vinogradenko
Category:
Middleware
Target version:
Estimated time:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Description

Currently the backblaze b2 cloud sync task uses up Class C transactions at an alarming rate. In the past 2 days, nearly 1.5 million transactions have been consumed via the b2_list_file_names command. This translates to about ~$5 worth of transactions per backblaze's pricing structure, and can get pretty expensive rather quickly.

Please add an option in the task setup menu to configure the polling rate similar to what is available with synology's implementation as seen in this article.

https://help.backblaze.com/hc/en-us/articles/224378407-Why-am-I-Reaching-Class-C-Transaction-Caps-with-Synology-


Risk
The risk is low given that its an option that can be turned on/off in case it does not work well for some providers

Acceptance Criteria
Once this is done fast list attribute should be available for providers supporting it.

It will use fewer transactions (important if you pay for them)
It will use more memory. Rclone has to load the whole listing into memory.
It may be faster because it uses fewer transactions
It may be slower because it can’t be parallelized

A note should be done in documentation about it.


Related issues

Related to FreeNAS - Bug #37358: Add "Use --fast-list" field to new UI for supported cloud sync backendsReady for Testing
Related to FreeNAS - Feature #37368: Tooltips for cloud sync optionsClosed

Associated revisions

Revision 7915066d (diff)
Added by Vladimir Vinogradenko 9 months ago

feat(cloud_sync): Add --fast-list option

Ticket: #27364

History

#1 Updated by Jacob Crisman over 1 year ago

  • File debug-san01-20171221161659.txz added

#2 Updated by Dru Lavigne over 1 year ago

  • Category changed from 2 to OS
  • Assignee changed from Release Council to Vladimir Vinogradenko
  • Target version set to 11.3

#3 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Unscreened to Screened

Currently Cloud Syncs occur every N minutes/hours, depending on how you've set it up. Thus, we can't change any polling rate as you suggest because there is no polling.

What we can do is enable https://rclone.org/docs/#fast-list option (b2 provider supports it) that is promised to reduce number of listing files transactions.

#4 Updated by Jacob Crisman over 1 year ago

Ahh, wasn't aware that this was using rclone as its method. Fast-list is definately the way to go here. I was already doing our backblaze backup with rclone prior to 11.1 via a jail. Is it possible to set up a cron job to run the backup task with this option enabled as a workaround until the feature gets added?

#5 Updated by Vladimir Vinogradenko over 1 year ago

Yes, this will work. You may do ps ax | grep rclone when backup job is running to get our base command-line options and also a temporary config file path to start with.

#6 Avatar?id=14398&size=24x24 Updated by Kris Moore about 1 year ago

  • Status changed from Screened to Not Started

#7 Avatar?id=13649&size=24x24 Updated by Ben Gadd 12 months ago

  • Target version changed from 11.3 to Backlog

#8 Updated by Vladimir Vinogradenko 12 months ago

  • Severity set to Medium

#9 Updated by William Grzybowski 10 months ago

  • Category changed from OS to Middleware

#10 Updated by William Grzybowski 10 months ago

  • Subject changed from Add Backblaze polling rate option to Cloud Sync Tasks menu to Provide option to enable fast-list for supported backends
  • Description updated (diff)
  • Target version changed from Backlog to 11.3

#11 Updated by Vladimir Vinogradenko 9 months ago

  • Related to Bug #37358: Add "Use --fast-list" field to new UI for supported cloud sync backends added

#12 Updated by Vladimir Vinogradenko 9 months ago

  • Status changed from Not Started to In Progress

#13 Updated by Vladimir Vinogradenko 9 months ago

  • Status changed from In Progress to Ready for Testing

#14 Updated by Dru Lavigne 9 months ago

  • Subject changed from Provide option to enable fast-list for supported backends to Provide option to enable fast-list for supported cloud sync backends
  • Needs Merging changed from Yes to No

#15 Updated by Dru Lavigne 9 months ago

  • File deleted (debug-san01-20171221161659.txz)

#16 Updated by Dru Lavigne 6 months ago

#17 Updated by Dru Lavigne 3 months ago

  • Target version changed from 11.3 to 11.3-BETA1

#18 Updated by Dru Lavigne 3 months ago

  • Subject changed from Provide option to enable fast-list for supported cloud sync backends to Add option to middleware to enable fast-list for supported cloud sync backends

#19 Updated by Dru Lavigne 2 months ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No

#21 Updated by Dru Lavigne 24 days ago

  • Subject changed from Add option to middleware to enable fast-list for supported cloud sync backends to Add fast-list support to legacy UI for supported cloud sync backends

#22 Updated by Dru Lavigne 24 days ago

  • Status changed from Done to Ready for Testing
  • Needs QA changed from No to Yes
  • Needs Doc changed from No to Yes

Also available in: Atom PDF