# Quick Start - [Quick Start](#quick-start) - [Interface Overview](#interface-overview) - [Task Management Interface](#task-management-interface) - [Create Task Interface](#create-task-interface) - [Create Node Interface](#create-node-interface) - [Create Source Interface](#create-source-interface) - [Instructions](#instructions) - [Add Endpoint](#add-endpoint) - [Create Task](#create-task) - [Start Migration](#start-migration) - [Pause Migration](#pause-migration) - [Retry](#retry) - [Export Failed File List](#export-failed-file-list) - [Cache Explanation](#cache-explanation) - [Migration Type Overview](#migration-type-overview) - [S3](#s3) - [local](#local) - [http](#http) --- ## Interface Instructions After the service starts, access the US3SYNC interface by opening the page https://: in a browser, where you can perform data migration operations. ### Task Management Interface - **Task Operations:** - **Start Migration:** Begin file migration. - **Start Validation:** Perform file validation by comparing file sizes between source and destination directories, fetching full file lists from both. - **Manage - Retry:** Retry migration for files that failed in previous attempts. - **Manage - Export:** Export a list of failed files to the `.US3SYNC/details/[jobid]/error.log` location in the destination bucket. - **Manage - Delete:** Delete a task and clear its cache records. - **Manage - Clone:** Open the task creation interface with the current task's information pre-filled. - **Statistics:** ``` ================================= Total Files: Total number of files to migrate. Pending Files: Number of files waiting to migrate. In Progress: Number of files currently being migrated. Successful: Number of successfully migrated files. Failed: Number of files that failed to migrate. ================================= ``` - **Error Log Export:** If migration tasks fail, use the **Export** button in the task management interface to export error logs. The logs will be uploaded to US3 and a downloadable URL will be provided. You can download the logs using `wget` from the node machine and troubleshoot the issue by referring to the FAQ. --- ### Create Task Interface - **Task Type:** Choose the migration task type. - **Regex Match Keys:** Use regular expressions to filter the source list. - **Schedule Execution:** Configure periodic execution. - **Sync Deletion:** Remove files in the destination that do not exist in the source. - **Job ID:** Click the expand icon next to the task name to obtain the Job ID. --- ### Create Node Interface - **IP Address:** Enter the worker node's IP address. - **Username:** Enter the SSH username for the worker node. - **Password:** Enter the SSH password for the worker node (leave blank for password-less login). --- ### Create Source Interface - **Timeout:** Specify upload/download timeout (default is 60s). - **Domain:** Enter the bucket's domain suffix, e.g., `cn-bj.ufileos.com`. - **Prefix:** Use `PrefixFileList` to input multiple prefixes. --- ## Instructions ### Add Endpoint Add a migration source by clicking the **Create Endpoint** button. Fill in the required information in the dialog box and click **OK**. --- ### Create Task Create a migration task by clicking the **Create Task** button. Fill in the required information in the dialog box and click **OK**. To create a fetch task, select the **Fetch Task** option in the interface. To obtain the **Job ID**, click the expand icon next to the task name. **Migration Process:** - Files under the specified source prefix will be migrated to the specified destination prefix. Examples: ``` 1. Source Prefix: A/ | Destination Prefix: B/ All files under A/ will be migrated to B/. Example: Source file A/a.txt will become B/a.txt at the destination. 2. Source Prefix: A/ | Destination Prefix: (empty) All files under A/ will be migrated to the root directory of the destination. Example: Source file A/a.txt will become a.txt at the destination root. ``` **Overwrite Options:** - **Full Overwrite:** Force upload all files from the source to US3. - **Conditional Overwrite:** Migrate files based on specified criteria: - File type - File size - File timestamp (migrate files that are newer in the source). - **No Overwrite:** Skip files that already exist in the destination. **Scheduled Tasks:** - Configure migration to run periodically, specifying the interval and start time. **Incremental Filters:** - **Absolute Time Filter:** Specify a start time for file migration. - **Recent Time Filter:** Specify files modified in the last few hours. For periodic tasks, the filter adjusts based on the current time. **Regex Matching:** - If enabled, the source list will be filtered using the specified regex. - The regex does not match the source prefix, allowing finer control within a prefix. --- ### Add Alerts - Notifications are sent via POST requests for task completions or worker failures. - Collect error metrics for migration and fetch tasks. - Configure custom alert rules for task errors using string-based logic. Notifications are sent via POST requests upon rule triggers. **Example Alert Logic:** ``` (RESOURCE_ERR_4XX > 100 || RESOURCE_ERR_5XX > 200) && TARGET_ERR_4XX > 150 && Q_LEN > 150 ``` **Supported Variables:** - `RESOURCE_ERR_4XX`: Number of 4XX errors from the source. - `RESOURCE_ERR_5XX`: Number of 5XX errors from the source. - `TARGET_ERR_4XX`: Number of 4XX errors from the destination. - `TARGET_ERR_5XX`: Number of 5XX errors from the destination. - `LOCAL_ERR`: Number of other errors. - `Q_LEN`: Number of files in the fetch task queue. --- ### Start Migration Refer to the [Interface Overview](#interface-overview) and click the **Start Migration** button. --- ### Pause Migration Refer to the [Interface Overview](#interface-overview) and click the **Pause** button. --- ### Retry Retry migrating failed files by clicking **Manage - Retry** in the [Task Management Interface](#task-management-interface). --- ### Export Failed File List Export the list of failed files by clicking **Manage - Export** in the [Task Management Interface](#task-management-interface). --- ## Cache Explanation US3SYNC uses **Pika** to provide caching services. Cached data includes: - Task details - Migration statistics - Migration logs --- ## Migration Type Overview | Type | Description | |-------|-----------------------------------------------------------------------------| | S3 | Migration based on S3 interfaces, supporting any cloud platform with S3 API | | OSS | Migration based on OSS cloud storage interfaces | | Qiniu | Migration based on Qiniu cloud storage interfaces | | Youpai| Migration based on Youpai cloud storage interfaces | | US3 | Migration based on US3 cloud storage interfaces | | NAS | Migration from NAS systems or local files | | URL | Migration based on a list of URLs | --- ### S3 AWS S3 sources require an additional `region` field. Refer to: . For third-party S3 implementations, consult the respective documentation. --- ### local `local` sources require the directory path to be provided. --- ### http `http` sources require a URL list of files to be migrated. Each line in the list should represent a file URL: ``` ```