C:\\Users\\Administrator\\Downloads\\cos_migrate
unzip cos_migrate_tool_v5-master.zip && cd cos_migrate_tool_v5-master
COS_Migrate_tool|——conf #Directory of the configuration file| |——config.ini #Migration configuration file|——db #Store the record of successful migrations|——dep #JAR package complied by the main logic of the program|——log #Log generated during tool execution|——opbin #Script for compiling|——result #A directory used to save successful migration records. The record file is named "date.out". The format is "absolute path\\tfile size\\tlast modified time".|——src #Source code of the tool|——tmp #Temporary file storage directory|——.gitignore #Files and folders ignored by the Git version controller.|——pom.xml #Project configuration file|——README #Readme document|——start_migrate.sh #Migration startup script for Linux|——start_migrate.bat #Migration startup script for Windows
db
directory mainly records the IDs of files successfully migrated by the tool. Each migration job will first compare the records in the db
directory. If the ID of the current file has already been recorded, the current file will be skipped, otherwise it will be migrated.log
directory keeps all the logs generated during tool migration. If an error occurs during migration, first check error.log
in this directory../conf/config.ini
) first. This file contains the following parts:type=migrateLocal
.[migrateType]type=migrateLocal
# The common configuration section of the migration tool includes account information to be migrated to the destination COS.[common]secretId=COS_SECRETIDsecretKey=COS_SECRETKEYbucketName=examplebucket-1250000000region=ap-guangzhoustorageClass=StandardcosPath=/https=offtmpFolder=./tmpsmallFileThreshold=5242880smallFileExecutorNum=64bigFileExecutorNum=8entireFileMd5Attached=onexecuteTimeWindow=00:00,24:00outputFinishedFileFolder=./resultresume=falseskipSamePath=falserequestTryCount=5
Name | Description | Default Value |
secretId | SecretId of your key. Replace COS_SECRETID with your real key information, which can be obtained on the TencentCloud API key page in the CAM console | - |
secretKey | SecretKey of your key. Replace COS_SECRETKEY with your real key information, which can be obtained on the TencentCloud API key page in the CAM console | - |
bucketName | Name of the destination bucket in the format of <BucketName-APPID> . The bucket name must include the APPID such as examplebucket-1250000000 | - |
region | Region information of the destination bucket. For the region abbreviations in COS, see Regions and Access Domain Names | - |
storageClass | Storage class for the migrated data. Valid values: Standard , Standard_IA , Archive . For more information, see Storage Class Overview. | Standard |
cosPath | COS path to migrate to. / indicates to migrate to the root path of the bucket, /folder/doc/ indicates to migrate to /folder/doc/ in the bucket. If /folder/doc/ does not exist, a path will be created automatically | / |
https | Whether to transfer via HTTPS. on: Yes, off: No. It takes time to enable transfer via HTTPS, which is suitable for scenarios that demand high security. | off |
tmpFolder | The directory used to store temporary files when data is migrated from another cloud storage service to COS, which will be deleted after the migration is completed. The format must be an absolute path: The separator on Linux is /, such as /a/b/c The separator on Windows is \\, such as E:\\\\a\\\\b\\\\c The default value is the tmp directory in the path of the tool | ./tmp |
smallFileThreshold | Number of bytes as the threshold for small files. If the size is greater than or equal to this threshold, multipart upload is used; otherwise, simple upload is used. The default value is 5 MB(5242880 Byte) | 5242880 |
smallFileExecutorNum | Concurrency for uploading small files (smaller than smallFileThreshold) via simple upload. Decrease the concurrency if files are uploaded to COS via public network with low bandwidth | 64 |
bigFileExecutorNum | Concurrency for uploading large files (greater than or equal to smallFileThreshold) via multipart upload. Decrease the concurrency if files are uploaded to COS via public network with low bandwidth | 8 |
entireFileMd5Attached | The migration tool calculates the MD5 of the entire file and stores it in the custom header "x-cos-meta-md5" of the file for subsequent verification, because the ETag of a large file uploaded to COS via multipart upload is not the MD5 of the entire file | on |
executeTimeWindow | Execution time window, at a granularity of minutes. This parameter defines the daily execution time range of the migration tool. For example:
The parameter "03:30,21:00" means that tasks are executed between 03:30 and 21:00. Outside these hours, the migration tasks enter a sleep state. In sleep state, migration is paused but the progress is retained, and migration is automatically resumed at the next time window. Note that the end time point must be later than the start time. | 00:00,24:00 |
outputFinishedFileFolder | This directory stores the results of successful migration tasks, and result files are named by date, for example, ./result/2021-05-27.out , where ./result is the directory that is created. Each line in the result files is in the format of "Absolute path"\\t"File size"\\t"Last modified time" . If outputFinishedFileFolder is left empty, no results will be output. | ./result |
resume | Whether to continue with the result of the last run and traverse through the list of files from the source. The tool starts from scratch by default. | false |
skipSamePath | Whether to skip the current file if a file with the same name already exists in COS. By default, the tool does not skip the current file: it overwrites the existing file. | false |
requestTryCount | Total number of attempts for each file upload. | 5 |
# Configuration section for migration from a local system to COS[migrateLocal]localPath=E:\\\\code\\\\java\\\\workspace\\\\cos_migrate_tool\\\\test_dataexcludes=ignoreModifiedTimeLessThanSeconds=
Configuration Item | Description |
localPath | Absolute path of the local directory Linux uses a slash (/) as the delimiter, for example, /a/b/c .Windows uses two backlashes (\\) as the delimiter, for example, E:\\\\a\\\\b\\\\c .Note: You can enter only a directory path but not file path for this parameter; otherwise, an error will occur while parsing the target object name. In the case of cosPath=/ , the request will be incorrectly parsed into a bucket creation request. |
excludes | Absolute path of the directory or file to be excluded, meaning some directories or files under localPath are not to be migrated. Multiple absolute paths are separated by semicolons. If this is left blank, all files in localPath will be migrated |
ignoreModifiedTimeLessThanSeconds | Exclude files that have an update time less than a certain period of time from the current time (in seconds). This item is left blank by default, indicating files are not to be filtered by the time specified by lastmodified . It is suitable for scenarios where you run the migration tool while updating files and don't want files being updated to be migrated to COS. For example, if it is configured as 300 , only files updated at least 5 minutes ago will be uploaded. |
config.ini
file by running the following command:sh start_migrate.sh
sh start_migrate.sh -Dcommon.cosPath=/savepoint0403_10/
db
directories. Refer to db
information in the tool structure above.sectionName
is the section name of the configuration file. sectionKey
is the name of the configuration item in the section. sectionValue
is the value of the configuration item in the section. COS path to which data is migrated to should be in the format of -Dcommon.cosPath=/bbb/ddd.
Was this page helpful?