tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Object Storage
    Documentation
    Download PDF

    Migrating Data from URL to COS

    Last updated: 2024-03-25 15:11:17
    Download PDF

      Background

      If you want to migrate data using a URL list as the data source address, COS supports the following three ways:
      Migration Method
      Description
      Migration Service Platform (MSP)
      MSP is a platform that integrates multiple migration tools and provides visual interfaces to help you monitor and manage large-scale data migration jobs with ease. The File Migration Tool on it can help you migrate data from various public clouds or data origin servers to COS.
      COS origin-pull automatically migrates data with read/write access requests from the data origin server to COS. This can help you separate hot data from cold data quickly and speed up reads/writes of hot data in your business system.
      COS Migration is an all-in-one tool integrating COS data migration features. You can migrate data to COS quickly after completing simple configurations.
      
      In the process of migrating data from the origin server to the cloud, if you only want to migrate hot data while leaving the cold data in the origin server, you can use COS origin-pull, which migrates the hot data accessed by read/write requests to COS and automatically separates the business data with low access frequency.

      Migration Practice

      Migration Service Platform (MSP)

      The steps are as follows:
      1. Log in to MSP.
      2. Click Migration Tool on the left sidebar, locate File Migration Tool, and click Use Now.
      3. Create a migration task and configure task information.
      4. Start the task.
      For more information, please see Migrating from a URL List.

      Tips

      During data migration, how fast the data source can be read from depends on the network connection, and selecting a higher QPS concurrence value when creating a file migration task will help speed up the migration.

      COS origin-pull

      The steps are as follows:
      1. Log in to the COS Console and enable the origin-pull setting in the destination bucket.
      2. Configure the bucket origin-pull address and save the configuration.
      3. Redirect the read/write requests from your business system to COS.
      For more information, please see Setting Origin-Pull.

      Tips

      The following steps demonstrate how to separate hot data and cold data. Hot data can be seamlessly migrated to COS for faster reads.
      1. Redirect the read/write requests in your business system to COS, enable the origin-pull setting in the COS Console, and configure the data origin server as the source address. The system structure is as shown below.
      
      
      2. After a period of time, the cold data remains in the origin server, while the hot data has been migrated to COS without any impact on your business system during migration.

      COS Migration

      The steps are as follows:
      1. Install the Java environment.
      2. Install the COS Migration tools.
      3. Modify the configuration file.
      4. Launch the tool.
      For more information, please see COS Migration Tool.

      Tips

      Below describes how to configure COS Migration to maximize the migration speed:
      1. Adjust the threshold and concurrence of large/small files based on your network environment to implement optimal migration where large files are split into multiple parts and small files are transferred concurrently. Adjust the tool execution time and set a bandwidth limit to ensure that bandwidth usage by data migration will not affect your business operations. Such adjustments can be made by modifying the following parameters in the [common] section of the config.ini configuration file:
      Parameter Name
      Parameter Description
      smallFileThreshold
      Threshold for small files. If the size of a file is higher than or equal to this threshold, multipart upload will be used; The default value is 5 MB.
      bigFileExecutorNum
      Concurrence of large files, which is 8 by default. If COS is connected to over the public network with low bandwidth, reduce this value.
      smallFileExecutorNum
      Concurrence of small files, which is 64 by default.If COS is connected to over the public network with low bandwidth, reduce this value.
      executeTimeWindow
      This parameter defines the time range when the migration tool will execute a task every day, which will enter sleep mode at other times. In sleep mode, the migration will be paused and the migration progress will be retained until the next time window when the migration will be resumed automatically.
      2. Distributed parallel transfer can further speed up the migration. You can install COS Migration on multiple servers and perform migration tasks separately for different data sources.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy