Add option to middleware to enable fast-list for supported cloud sync backends
Currently the backblaze b2 cloud sync task uses up Class C transactions at an alarming rate. In the past 2 days, nearly 1.5 million transactions have been consumed via the b2_list_file_names command. This translates to about ~$5 worth of transactions per backblaze's pricing structure, and can get pretty expensive rather quickly.
Please add an option in the task setup menu to configure the polling rate similar to what is available with synology's implementation as seen in this article.
The risk is low given that its an option that can be turned on/off in case it does not work well for some providers
Once this is done fast list attribute should be available for providers supporting it.
It will use fewer transactions (important if you pay for them) It will use more memory. Rclone has to load the whole listing into memory. It may be faster because it uses fewer transactions It may be slower because it can’t be parallelized
A note should be done in documentation about it.
#3 Updated by Vladimir Vinogradenko about 1 year ago
- Status changed from Unscreened to Screened
Currently Cloud Syncs occur every N minutes/hours, depending on how you've set it up. Thus, we can't change any polling rate as you suggest because there is no polling.
What we can do is enable https://rclone.org/docs/#fast-list option (b2 provider supports it) that is promised to reduce number of listing files transactions.
#4 Updated by Jacob Crisman about 1 year ago
Ahh, wasn't aware that this was using rclone as its method. Fast-list is definately the way to go here. I was already doing our backblaze backup with rclone prior to 11.1 via a jail. Is it possible to set up a cron job to run the backup task with this option enabled as a workaround until the feature gets added?