Project

General

Profile

Feature #34735

Support all cloud sync services that rclone supports in new UI

Added by Dru Lavigne over 1 year ago. Updated about 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
Lola Yang
Category:
GUI (new)
Target version:
Estimated time:
Severity:
Med High
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Related issues

Has duplicate FreeNAS - Feature #35107: Allow passing auxiliary arguments to rcloneClosed
Copied from FreeNAS - Feature #33441: Support all cloud sync services that rclone supportsDone

History

#1 Updated by Dru Lavigne over 1 year ago

  • Copied from Feature #33441: Support all cloud sync services that rclone supports added

#2 Updated by Erin Clark over 1 year ago

  • Assignee changed from Erin Clark to Lola Yang

#3 Updated by Dru Lavigne over 1 year ago

  • Target version changed from 11.2-RC2 to 11.2-BETA1

Lola: this one is needed for feature parity.

#4 Updated by Vladimir Vinogradenko over 1 year ago

Now we don't need to support each cloud sync provider in the UI. UI should render cloud sync credentials and cloud sync tasks form using declarative form description for chosen provider obtained from backend.

New method cloudsync.providers returns list of all supported cloud sync providers. Example:

[
    {
        # Provider ID
        "name": "S3",
        # Provider display name
        "title": "Amazon S3",
        # List of fields (credentials `attributes`)
        "credentials_schema": [
            {
                # Key
                "property": "access_key_id",
                "schema": {
                    "title": "Access Key ID",
                    "_required_": true,
                    "type": "string" 
                }
            },
            {
                "property": "secret_access_key",
                "schema": {
                    "title": "Secret Access Key",
                    "_required_": true,
                    "type": "string" 
                }
            },
            {
                "property": "endpoint",
                "schema": {
                    "title": "Endpoint URL",
                    "_required_": false,
                    "type": [
                        "string",
                        "null" 
                    ]
                }
            }
        ],
        # If this provider uses buckets
        "buckets": true,
        # List of fields (cloud sync task `attributes`)
        "task_schema": [
            {
                "property": "encryption",
                "schema": {
                    "title": "encryption",
                    "_required_": false,
                    "type": [
                        "string",
                        "null" 
                    ],
                    "enum": [
                        null,
                        "AES256" 
                    ]
                }
            }
        ]
    },
]

#5 Updated by Vladimir Vinogradenko over 1 year ago

New plugin for managing cloud sync credentials is cloudsync.credentials. It's interface is same as legacy backup.credentials

#6 Updated by Vladimir Vinogradenko over 1 year ago

New plugin for managing cloud sync tasks is cloudsync. It's interface is almost the same as legacy backup

Changes:

  1. Schedule-related fields are grouped in schedule field (same as in cron plugin)
  1. Credentials ID field is credentials not credential
  1. Attributes do not include bucket field if provider does not use buckets

#7 Updated by Vladimir Vinogradenko over 1 year ago

Buckets are listed using universal method cloudsync.list_buckets with one parameter: Credentials ID

#8 Updated by Vladimir Vinogradenko over 1 year ago

This all is a huge work so it probably should be retargeted for 11.2-U1. Meanwhile legacy backup plugin still works.

#9 Updated by Lola Yang over 1 year ago

  • Status changed from Unscreened to In Progress

#10 Updated by Dru Lavigne over 1 year ago

  • Target version changed from 11.2-BETA1 to 11.2-U2

#12 Updated by Lola Yang over 1 year ago

  • Related to Feature #35107: Allow passing auxiliary arguments to rclone added

#13 Updated by Dru Lavigne over 1 year ago

  • Related to deleted (Feature #35107: Allow passing auxiliary arguments to rclone)

#14 Updated by Dru Lavigne over 1 year ago

  • Has duplicate Feature #35107: Allow passing auxiliary arguments to rclone added

#15 Updated by Dru Lavigne over 1 year ago

  • Target version changed from 11.2-U2 to 11.2-BETA2

#16 Updated by Dru Lavigne over 1 year ago

  • Subject changed from Support all cloud sync services that rclone supports to Support all cloud sync services that rclone supports in new UI

#17 Updated by Vladimir Vinogradenko over 1 year ago

Please remove Auxiliary arguments from everywhere, it's been decided that users should not see this until https://redmine.ixsystems.com/issues/36633 is implemented

#18 Updated by Lola Yang over 1 year ago

Vladimir Vinogradenko wrote:

Please remove Auxiliary arguments from everywhere, it's been decided that users should not see this until https://redmine.ixsystems.com/issues/36633 is implemented

That field removed from UI, but will send an empty string to middleware to avoid trace back.

#19 Updated by Vladimir Vinogradenko over 1 year ago

Lola, when updating cloud sync, it's important to send current value because it can still be modified using CLI and should be preserved when using web UI.

#20 Updated by Lola Yang over 1 year ago

Vladimir Vinogradenko wrote:

Lola, when updating cloud sync, it's important to send current value because it can still be modified using CLI and should be preserved when using web UI.

Yes. when updating cloud sync, user can not modified the 'args' field via UI, so it will keep the original value (empty or the value modified via CLI).

#21 Updated by Lola Yang over 1 year ago

  • Status changed from In Progress to Ready for Testing

The PR already merged to master.

#22 Updated by Timothy Moore II over 1 year ago

  • Status changed from Ready for Testing to Passed Testing
  • Needs QA changed from Yes to No
  • Needs Merging changed from Yes to No

Testing with FreeNAS Mini/System updated to FreeNAS-11.2-MASTER-201807160837:

Go to System/Cloud Credentials/Add. Open “Provider” drop-down menu. These options are available: Amazon Cloud Drive, Amazon S3, Backblaze B2, Box, Dropbox, FTP, Google Cloud Storage, Google Drive, HTTP, Hubic, Mega, Microsoft Azure Blob Storage, Microsoft OneDrive, pCloud, SFTP, WebDAV, and Yandex. Switch to legacy UI and go to the same form. Verify options and functionality are the same.

Switch back to new UI.

Go to Tasks/Cloud Sync Tasks/Add. Add sync task with existing B2 credential. Verify “Bucket” properly detects and lists the available buckets. Verify “Schedule” options match the interface from Tasks/Cron Jobs/Add.

Create an HTTP Cloud Credential and retest. Verify “Bucket” field is not visible. Verify other fields function as expected.

#23 Updated by Dru Lavigne over 1 year ago

  • Needs Doc changed from No to Yes

#24 Updated by Dru Lavigne over 1 year ago

  • Status changed from Passed Testing to Done
  • Needs Doc changed from Yes to No

#25 Updated by John Hennessy about 1 year ago

  • Status changed from Done to Broken

I believe there is an issue with the Cloud Credentials UI for the Google Drive provider. While Name, Provider and Access Token fields are all accepted, when the Cloud Sync Task is run the sync does not run. The new UI simply returns "The cloud sync task has started" but the legacy UI shows "FAILED: rclone failed". The log for that attempt shows:

"2018/08/31 11:25:36 Failed to create file system for "remote:": drive: failed when making oauth client: failed to create oauth client: invalid character 'U' looking for beginning of value"

The invalid character 'U' reported is always the first character of what is entered in the Cloud Credential Access Token field.

running rclone from the shell with the same values works correctly.

Using the UI to enter value for Google Cloud sync works correctly.

#26 Updated by Dru Lavigne about 1 year ago

  • Status changed from Broken to Done

John: please create a new ticket that includes your debug after reproducing the issue. Thanks!

Also available in: Atom PDF