This document describes how to configure an Agent semi-managed migration task to migrate data from AWS S3 (International) to Tencent Cloud COS.
Note:
You need to log in to the AWS Management Console (International) for migration from AWS S3 (International).
Preparations
AWS S3
Create an AWS IAM account and grant the relevant permissions. Below is a simple guide; for detailed instructions, please refer to AWS S3 Product Documentation.
1. Log in to the AWS Console.
2. In the navigation pane, select Users and then Add User. Type in the username for the new user.
3. Select the access permission type that this group of users will have. Choose Programmatic access and AWS Management Console access.
4. Click Next: Permissions(Next: Permissions). On the Set permissions page, specify how you want to assign permissions to the new user. Grant IAM account storage space read-write permissions.
5. Click Create user. To view the user's access keys (Access Key ID and Secret Access Key), select show next to each password and access key you wish to view. To save the access keys, select download .csv, to obtain AccessKeyID and AccessKeySecret.
Tencent Cloud COS
1. Create target bucket:
Create a destination bucket to store the migrated data. For more details, see Creating Bucket.
2.2 Choose Users > User List in the left sidebar to enter the user list page.
2.3 Create a sub-user, and check the Programming access and Tencent Cloud console access options.
2.4 Search for and check the QcloudMSPFullAccess and QcloudCOSAccessForMSPRole policies.
2.5 Complete the sub-user creation and save the username, login password, SecretId, and SecretKey.
Note:
You can also log in to Cloud Migration with your root account. However, for security reasons, we recommend that you create a sub-account, use the sub-account's API key for migration, and delete the sub-account after migration.
2. Click Object Storage Migration in the left sidebar to enter the object storage migration page.
Creating Migration Task
1. On the object storage migration page, click Create task to enter the migration task configuration page and set the migration parameters.
2. Specify the migration task name.
Task name: 1 to 60 characters ([a-z], [A-Z], [0-9] and [_]). You can use this name to view the migration status and progress in the task list.
3. Task size estimation. Please accurately fill in the task size so that Tencent Cloud can better prepare resources. This field is optional.
4. Set the file source for migration.
Service Provider: Select AWS S3 (International) for this field.
AccessKey, SecretKey: Enter the AccessKey and SecretKey of the account used for migration in the text boxes.
Bucket name: Upon entering the key, click the Refresh button to the right of the "Bucket name" dropdown menu to retrieve a list of source object storage buckets. Alternatively, you may opt to manually input the name of the source bucket.
Header migration mode: If the file in the source bucket has headers and tags that need to be retained after migration, please select the retention option or set the replacement rule.
File name filtering rules: Choose to migrate all the files of the specified bucket or migrate only files with the specified prefix.
Time range: Specify the time range. Migrate files that are added or modified within this time period.
Execution Speed, Speed Limit Mode: All public cloud object storage services have speed limits. To ensure business stability, please confirm with the service provider and set the maximum available migration bandwidth in Mbps before migrating.
5. Select the target location to migrate to.
Service Provider: The default value is Tencent Cloud COS.
SecretId, SecretKey: Enter the SecretId and SecretKey of the sub-user for migration.
Bucket name: After you enter the Key, click Refresh on the right side of the "Bucket Name" drop-down box to view the list of destination COS buckets.
Storage Path: Specify the directory in the destination bucket for migration.
Save to the root directory: Store the files in the source bucket to the root directory of the destination bucket according to the original relative paths.
Save to the specified directory: Store the files in the source bucket to the specified directory according to the original relative paths.
For example:
For files /a.txt and /dir/b.txt in the source bucket, if "dest" is filled in this field, the paths of the two files in the destination bucket will be /dest/a.txt and /dest/dir/b.txt after migration.
If dest/20180901 is filled in this field, the paths of the two files in the destination bucket will be /dest/20180901/a.txt and /dest/20180901/dir/b.txt after migration.
File with the same name: Specify the handling method for files with the same name.
Notes:
If the method is Overwrite, files in the destination bucket will be overwritten directly during migration.
If the method is Skip, judgment will be performed based on the last modification time (LastModified). Specifically:
If the last modification time of the file in the source bucket is later than or the same as that of the file in the destination bucket, then overwrite the file in the destination bucket.
If the last modification time of the file in the source bucket is earlier than that of the file in the destination bucket, skip the file in the source bucket.
Secondary migration is required if the object (file) content is changed during migration.
Storage type: Set the storage class for migrated files according to the migration requirements. Available options: Use standard storage for all, Use Standard Infrequent Access Storage for all, Keep the original storage properties, and Save all as archive storage.
6. Select migration pattern.
Mode Selection:
Create a migration task and start fully-managed migration immediately: When fully-managed migration is used, Cloud Migration will access the source bucket via the public network for migration after you click "Create and Start".
Create a migration task and download the Agent manually to start migration: When semi-managed migration is used, only task creation is completed after you click "Create and Start". You need to manually download the Agent and deploy it on the origin server before you can start the migration. For Agent deployment, see Using Semi-Managed Migration Agent.
Scheduled Task: Scheduled tasks can be executed repeatedly to synchronize incremental files in the source bucket. Such tasks are executed immediately for the first time and can be triggered later either after a specified interval or according to the scheduling rules set in Custom UNIX cron format. When a scheduled task is executed again, files with the same name as those in the destination bucket will be skipped. In other words, existing files with the same name in the destination bucket are retained, which means that only incremental files are synchronized.
7. Click Create and start to start the migration task.
Viewing Migration Status and Progress
You can view the status and progress of all file migration tasks on the Object Storage Migration page:
For completed tasks, the message in green indicates the task is completed with all files successfully migrated, while the message in yellow indicates the migration task is completed but some files failed to be migrated.
You can click Retry failed task to retry the migration of failed files involved in the task. Files that have been successfully migrated will not be retried.
You can click Export list of failed files to export the list of files that failed to be migrated.
Estimating File Migration Time
The migration speed is subject to the lowest speed at each stage of the migration process and is affected by the network transmission speed and maximum concurrency level. For more details, see Estimating File Migration Time.
Yes
No
Was this page helpful?