tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Data Transfer Service
    Documentation
    Download PDF

    Migration Operation Guide

    Last updated: 2024-09-24 15:33:35
    Download PDF

      Operation Scenes

      DTS-based Redis data migration supports full + incremental data migration, allowing both the historical data in the source database before the migration and any new data written during the migration process to be migrated together.
      This document introduces how to use the DTS data migration feature to migrate data from Redis to TencentDB for Redis.

      Preparations

      1. Establish the access channel between DTS and the database in advance according to the type of access you need. For detailed instructions, see Network Preparation Work.
      IDC self-built databases/Databases from other cloud providers: The available access methods include Public Network/Direct Connect/VPN Access/CCN".
      When public network instances are migrated, ensure that the source instance is accessible over the public network and that the connection remains stable. Network fluctuations or failures can cause the migration to fail, and if that happens, you need to restart the migration task.
      Self-built databases on CVM: Select Self-Build on CVM as the access method.
      TencentDB Instance: Select Database as the access method.
      2. Pre-Migration Check
      You should check and make sure that the following items are passed before the migration; otherwise, the migration may fail.
      2.1. Big keys in the source database
      They may cause the buffer (client-output-buffer-limit) to overflow during the migration, leading to a migration failure.
      For TencentDB databases, you can use the performance optimization feature of TencentDB for DBbrain to quickly analyze big keys. For more information, see Memory Analysis.
      For non-TencentDB databases, use RDBTools to analyze big keys in Redis.
      Evaluate large keys for splitting or cleaning. If you need to retain them, set the source buffer size (client-output-buffer-limit) to infinite.
      config set client-output-buffer-limit 'slave 0 0 0'
      2.2. Limit on the number of TCP connections in the source Linux kernel
      If the number of concurrent business requests is high, check the limit on the number of connections in the Linux kernel before the migration. If this value is exceeded, the Linux server will actively disconnect from DTS.
      echo "net.ipv4.tcp_max_syn_backlog=4096" >> /etc/sysctl.conf
      echo "net.core.somaxconn=4096" >> /etc/sysctl.conf
      echo "net.ipv4.tcp_abort_on_overflow=0" /etc/sysctl.conf
      sysctl -p
      2.3. Access permission of the source RDB file directory
      Before the migration, check and make sure that the directory where RDB files are stored in the source database is readable; otherwise, the migration will fail.
      If the RDB file directory is not readable, run the following command in the source database to set "diskless replication". Then, RDB files will be directly sent to DTS for storage, with no need to be stored in the source database first and then sent.
      config set repl-diskless-sync yes
      2.4. (Optional) For scenes involving migration from a standard architecture to a cluster architecture, check for command compatibility issues. For detailed static and dynamic evaluations, see Standard to Clustered Architecture Migration Assessment.

      Directions

      Step 1:Create a migration task

      1. log in to DTS Console, select Data Migration page, then on the right side click Create Migration Task.
      2. On the new migration task page, select the task creation parameters, and then click Buy Now.
      Configuration Parameter
      Description
      Service Type
      Select Data Migration.
      Creation Mode
      Create a task: Create a completely new task.
      Create a similar task: Quickly create a task with the same configuration as that of a previous task. The new task's pre-filled options such as database type, access method, billing mode, and migration type are consistent with that of the historical task. Users can modify the settings if needed.
      Billing Mode
      Only pay-as-you-go is supported.
      Source Instance Type
      Select the source database type, which cannot be changed after purchase. Here, select Redis.
      Source Instance Region
      Select the source database region. If the source database is a self-built one, select a region nearest to it.
      Target Instance Type
      Select the target database type, which cannot be changed after purchase. Here, select Redis.
      Target Instance Region
      Select the target database region.
      Version
      Currently, only NewDTS is supported.
      Specification
      Currently, only the Xlarge specification is supported.
      Tag
      Set tag keys and values for the migration task to efficiently manage multiple tasks using tags.
      Task Name
      Select Name After Creation: The default task name will be the same as the task ID. You can rename the task after the migration task is created.
      Select Name Now: Enter the task name in the input box below.
      Terms of service
      Make sure to check the box for I have read and agree to <Cloud Database Service Terms>.
      Quantity
      You can purchase up to 10 migration tasks at a time.
      3. After the purchase is completed, the page will automatically redirect to the Data Migration Task List.

      Step 2:Set source and target databases

      1. In the Data Migration Task List, find the task you just created, and click Configure in the Operation column.
      2. Go to the Set source and target databases tab, fill in the source and target database information,and then click Test Connectivity. Once the test passes, click Save to proceed to the next step.
      Task Configuration
      Configuration Parameter
      Description
      Task Name
      Set a task name that is easy to identify.
      Running Mode
      Immediate execution: The task will start immediately after the pre-check validation passes.
      Scheduled execution: Set a specific time for the task to start. The task will not start immediately after the pre-check validation passes; instead, it will start at the scheduled time.
      Automatic Retry
      If it is selected, the migration task will automatically retry within the specified time range in case of task interruptions caused by network exceptions or similar issues. No manual intervention is required. The retry mechanism restarts the data migration from the beginning. During the retry process, any data that was previously migrated to the target end will either be cleared or overwritten.
      Source Database Settings
      Configuration Parameter
      Description
      Source Database Type
      The source database type selected during purchase, which cannot be changed.
      Service Provider
      If the source database is a self-built database on IDC or CVM, or a TencentDB database, select "Others".
      If the source database is an Alibaba Cloud database, Huawei Cloud, etc., select "Others".
      If the source database is an AWS database, select AWS. Currently, only AWS ElasticCache for Redis is supported; other AWS Redis databases are not supported. Depending on whether the PSYNC command is prefixed, there are different selections.
      For the PSYNC command with a prefix, select AWS.
      For the PSYNC command without a prefix, select Others.
      Region
      The region selected for the source database at the time of purchase cannot be modified.
      Access Type
      Select the network type through which the source database will connect to Tencent Cloud. Different access types require different network connectivity operations. For details, see Preparation Overview.
      If the source database is an IDC self-built database or a database from another cloud providers, the access methods available include Public Network/Direct Connect/VPN Access/CCN.
      Public Network: The source database can be accessed through a public IP.
      Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
      VPN Access: The source database can be interconnected with VPCs through VPN Connections.
      CCN: The source database can be interconnected with VPCs through CCN.
      If the source database is self-built on CVM, select Self-Build on CVM as the access method.
      If the source database is a TencentDB instance, select Database as the access method.
      Node Type
      Access Type: This parameter is displayed when an access type other than"Database" is selected. Choose according to your specific situation.
      Single-node migration: Refers to a cluster architecture with one primary node and multiple replicas.
      Cluster migration: Refers to a cluster architecture consisting of multiple shards, with each shard having several replica nodes. When cluster migration is selected, you will need to fill in the Node Info.
      Enter the addresses and passwords for all shard nodes in the source database cluster (IP:port: password or IP:port). For multiple nodes, enter each on a new line.
      If the source database is AWS, you need to enter the PSYNC name (IP:port:PSYNC name: password or IP:port:PSYNC name). The PSYNC name is the prefix used in the PSYNC command and should be requested from AWS. If the PSYNC command is not prefixed, it is necessary to set the preceding Service Provider parameter to Others and perform the migration as per an ordinary cluster.
      It is recommended to use the replica nodes (secondary nodes) of the source database. Migrating from the replica nodes can help avoid impacting the source database's business access.
      Public Network
      When Public Network is selected as the Access Type, the following parameters need to be configured.
      Host Address: The IP address or domain name of the source database.
      Port: The port used by the source database.
      Password: The password for the migration account of the source database.
      Self-Build on CVM
      When Self-Build on CVM is selected as the Access Type, the following parameters need to be configured.
      CVM instance: The instance ID of the CVM.
      Port: The port used by the source database.
      Password: The password for the migration account of the source database.
      Direct Connect
      When Direct Connect is selected as the Access Type, the following parameters need to be configured.
      VPC Direct Connect Gateway: Only VPC Direct Connect gateways are supported for Direct Connect. Ensure that the gateway is associated with the correct network type.
      VPC: Select the VPC and subnet.
      Host Address: The IP address of the source database.
      Port: The port used by the source database.
      Password: The password for the migration account of the source database.
      VPN Access
      When VPN Access is selected as the Access Type, the following parameters need to be configured.
      VPN Gateway: VPN Gateway. Select the VPN gateway instance that connects to the source database network.
      VPC: Select the VPC and subnet.
      Host Address: The IP address of the source database.
      Port: The port used by the source database.
      Password: The password for the migration account of the source database.
      Database
      When Database is selected as the Access Type, the following parameters need to be configured.
      Cross-account
      Same account: The source database instance and the target database instance belong to the same Tencent Cloud account.
      Cross-account: The source database and the target database instances belong to different Tencent Cloud accounts. For detailed instructions on cross-account operations, see Cloud Database Migration Across Accounts.
      Cloud Database Instance: Select the instance ID of the source database.
      Password: The password for the migration account of the source database.
      CCN
      When the Access Type is set to CCN, the following parameters need to be configured.
      When CCN is used for connection, it supports both same-account and cross-account CCN. Due to the complexity of network configurations, see Migrate Self-built Database to Tencent Cloud Database via CCN.
      Host network environment: Select according to your actual situation.
      If the source database is a TencentDB instance, select Tencent Cloud; if the source database is a self-built IDC database, select Self-built IDC; if the source database is from another cloud provider, select the corresponding network.
      Host Address: The IP address of the source database host.
      Port: The port used by the source database.
      Password: The password for the migration account of the source database.
      CCN instance account
      My Account: The CCN resources and DTS belong to the same Tencent Cloud root account.
      Other Account: The CCN resources and DTS belong to different Tencent Cloud root accounts.
      VPC CCN : The name of the CCN instance.
      VPC access: Select the VPC and subnet to access. The Access VPC refers to the VPC that connects to the DTS migration linkage within the CCN. Select a VPC other than the one to which the source database belongs from all the VPCs associated with the CCN.
      VPC access region: The region of the source database selected at the time of task purchase should match the region of the VPC you are connecting to. If they do not match, DTS will automatically change the region of the source database selected during task purchase to match the VPC region.
      Target Database Settings
      The target database parameter settings are similar to those of the source database. Only the differences are described below.
      Configuration Parameter
      Description
      Access Type
      Select based on your scene. The configuration principles for different access types are similar to those for the source database.
      Downward migration is supported. When the Access Type of the source database is set to Database, the target database can be selected as Public Network/Self-Build on CVM/Direct Connect/VPN Access/ CCN.
      Node Type
      This parameter appears when any Access Type other than "Database" is selected.
      Select based on the actual situation of the target database. Supported options include Single-node Migration, Redis Cluster Migration, and Proxy Cluster Migration.

      Step 3: Configuring Migration Options and Selecting Migration Objects

      In the Set migration options and select migration objects tab, configure the parameters and then click Next.
      Basic Options
      Configuration Item
      Description
      Migration Type
      The default is full + incremental migration and cannot be modified. Full + incremental migration means migrating the historical data of the source database before the migration and the new data written to the source database during the migration.
      Migration Object
      The default is to select the entire instance and cannot be modified.
      Target Database Writing Mode
      Clear Target Instance: When the source database data is synchronized to the target database, the existing data in the target database will be cleared and then rewritten.
      Overwrite: When the source database data is synchronized to the target database, the keys from the source database will overwrite the keys in the target database.
      Enable Target Database Read/Write
      Allow DTS or other services to write data to the target database. This parameter needs to be enabled when using the task retry feature.
      Automatic expiry of keys
      When it is selected, expired keys in the target database will be automatically purged. This may result in inconsistencies between the data in the source and target databases, so proceed with caution.
      Advanced Options
      Configuration Item
      Description
      Limit Data Transfer Rate
      This generally does not need to be set. However, if the DTS transfer rate is impacting the load on the target database, you can set a rate limit.
      Number of shards in the target database: If the target database is a TencentDB instance, the number of shards is automatically retrieved and cannot be modified.
      Traffic limit per shard in the target database: Enter the traffic limit per shard, with the setting range between 0-100 MB/s.
      If the target instance has a cluster architecture, then the maximum traffic per second for a single instance = number of shards * If the target instance has a cluster architecture, the maximum traffic per second for the entire instance is calculated as: number of shards x traffic limit per shard (MB/s).

      Step 4: Validating and Starting the Task

      1. On the task verification page, the system will automatically perform a pre-check. After the validation is successful, click Start Task to immediately enable the data migration. (If a scheduled execution time was set during task configuration, you can also select Start Later, and the task will be enabled at the scheduled time.)
      Note:
      Failed: It indicates that a check item fails and the task is blocked. You need to fix the problem and run the verification task again.
      Alarm: It indicates that a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. You need to assess whether to ignore the alarm or fix the problem and continue the task based on the alarm message.
      2. Return to the data migration task list, and you can see that the task has entered the Preparing status. After 1–2 minutes, the data migration task will be started.

      Step 5: Ending the Task

      1. If you need to view task progress details, delete the task, or perform other operations, click the corresponding button in the Operation column. For more details, see Task Management.
      2. When the task status is Ready to complete, it indicates that the incremental synchronization is finished, and you can end the task. Click Complete in the Operation column to end the migration task.
      It is recommended to end the task when the data difference between the target and source databases is 0 KB, and the time difference is 0 seconds.
      If you need to perform a business cutover after the migration is completed, see Cutover Instructions.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy