tencent cloud

Feedback

Operation Guide

Last updated: 2024-09-11 12:46:02

    Operation Scenarios

    DTS-based Redis data migration supports both full and incremental data migration, allowing you to migrate the source database's historical data as well as any new data written to the source during the migration process.
    This document provides guidance on using the DTS data migration feature to migrate data from Redis to TencentDB for KeeWiDB.

    Preparations

    1. Establish the access channel between DTS and the database in advance according to the access type you intend to use. For detailed instructions, see Network Preparation Work.
    IDC Self-hosted database/Other cloud provider database: The available access methods include Public Network/Direct Connect/VPN Access/CCN.
    When migrating an external instance, you should ensure that the source instance is accessible in the public network environment and that the connection remains stable. Any network fluctuations or failures may result in migration failure. If the migration fails, you will need to restart the migration task.
    Self-hosted database on CVM: Select Self-Build on CVM as the access method.
    TencentDB instance: Select Database as the access method.
    2. Pre-migration check
    The following checks need to be performed by the user before migration; otherwise, migration failure may occur.
    2.1 Check for large Keys on the source
    During migration, large keys may cause the client-output-buffer-limit buffer to overflow, leading to migration failure.
    For TencentDB instances, use the diagnostic and optimization features of TencentDB for DBbrain (DBbrain) to quickly analyze large keys. For detailed instructions, see Memory Analysis.
    For non-TencentDB databases, use rdbtools to analyze large Redis keys.
    Evaluate large keys and consider splitting or cleaning them. If you choose to retain large keys, set the source buffer size (client-output-buffer-limit) to unlimited.
    config set client-output-buffer-limit 'slave 0 0 0'
    2.2 Check the TCP connection limit of the source Linux Kernel
    If your business has a high number of concurrent requests, check the Linux kernel's connection limit before migration. If the number of business request connections exceeds the kernel's connection limit, the Linux server will proactively disconnect from DTS.
    echo "net.ipv4.tcp_max_syn_backlog=4096" >> /etc/sysctl.conf
    echo "net.core.somaxconn=4096" >> /etc/sysctl.conf
    echo "net.ipv4.tcp_abort_on_overflow=0" /etc/sysctl.conf
    sysctl -p
    2.3 Check access permissions for the source RDB file directory
    Before migration, ensure that the directory storing the RDB files on the source is readable. If the directory is not readable, the migration will fail due to the inability to access the RDB files.
    If the RDB file directory is not readable, execute the following command on the source to set up diskless replication, which allows the RDB file to be sent directly to DTS without needing to be stored on the source disk first.
    config set repl-diskless-sync yes
    2.4 (Optional) For scenes involving migration from a standard architecture to a cluster architecture, check for command compatibility issues. For details, see Check on Migration from Standard Architecture to Cluster Architecture to perform both static and dynamic evaluations.

    Operation Steps

    StepⅠ: Creating a Migration Task

    1. Log in to the DTS Console, navigate to the Data Migration page, and then click Create Migration Task on the right side.
    2. On the Create Migration Task page, select the task creation parameters, and then click Buy Now.
    Configuration Parameter
    Description
    Service Type
    Select Data Migration.
    Creation Mode
    Create a task: Create a brand new task.
    Create a similar task: Quickly create a task with the same configuration as a previous task. The new task will have pre-filled options consistent with the previous task, such as database type, access method, billing mode, and migration type. Users can modify these settings as needed based on their requirements.
    Billing Mode
    Only pay-as-you-go is supported.
    Source Instance Type
    Select the type of your source database. After purchase, this cannot be changed. Select Redis here.
    Source Instance Region
    Select the region where the source database is located. If the source is a self-hosted database, select the region closest to the self-hosted database.
    Target Instance Type
    Select the type of your target database. After purchase, this cannot be changed. Select KeeWiDB here.
    Target Instance Region
    Select the region where the target database is located.
    Version
    Currently, only NewDTS is supported.
    Specification
    Currently, only the Xlarge specification is supported.
    Tags
    Set tag keys and tag values for the migration task to efficiently manage multiple tasks using tags.
    Task Name
    Select Name After Creation: The task name will be the same as the Task ID by default. After the migration task is created, you can rename the task.
    Select Name Now: Enter the task name in the input box below.
    Terms and Conditions
    Make sure to check the box I have read and agree <Cloud Database Service Terms>.
    Quantity
    A maximum of 10 migration tasks can be purchased at one time.
    3. After the purchase is completed, the page will automatically redirect to the Data Migration Task List.

    Step Ⅱ: Configuring the Source and Target Databases

    1. In the Data Migration Task List, find the task you just created and click Configure in the Operation column.
    2. Enter the Set source and target databases tab, fill in the details for both the source and target databases, then click Test Connectivity. If the test passes, click Save to proceed to the next step.
    Task Configuration
    Configuration Parameter
    Description
    Task Name
    Set a name with business significance for easy task identification.
    Running Mode
    Immediate execution: The task will start immediately after the pre-check passes.
    Scheduled execution: Set a specific time for the task to start. The task will not start after the pre-check passes but will start at the scheduled time.
    Automatic Retry
    After this option is selected, if the migration task is interrupted due to network anomalies or other issues, DTS will automatically retry within the specified time range without requiring user intervention. The retry mechanism restarts the data migration from the beginning. When retrying, any data that was previously migrated to the target will be cleared or overwritten.
    Source Database Settings
    Configuration Parameter
    Description
    Source Database Type
    The selected source database type at the time of purchase cannot be modified.
    Service Provider
    If the source database is an IDC self-hosted database, a self-hosted database on CVM, or a TencentDB database, select Others.
    If the source database is Alibaba Cloud, Huawei Cloud, etc., select Others.
    Region
    The region selected for the source database at the time of purchase cannot be modified.
    Access Type
    Select the network type through which the source database will connect to Tencent Cloud. Different access types require different network configuration steps. For details, see Overview of Preparatory Work.
    If the source database is an IDC self-hosted database or a database from another cloud provider, the available access methods are Public Network/Direct Connect/VPN Access/CCN.
    Public network: The source database can be accessed via a public IP.
    Direct connect: The source database connects to Tencent Cloud Virtual Private Cloud via Direct Connect.
    VPN access: The source database connects to Tencent Cloud Virtual Private Cloud via VPN Connections.
    CCN: The source database connects to Tencent Cloud Virtual Private Cloud via Cloud Connect Network.
    If the source database is a self-hosted database on CVM, select Self-Hosted on CVM as the access method.
    If the source database is a TencentDB instance, select Database as the access method.
    Node Type
    This parameter is displayed when any Access Type other than Database is selected. Select according to your actual situation.
    Single node migration: Refers to a cluster architecture with one primary node and multiple replica nodes.
    Cluster migration: Refers to a cluster architecture composed of multiple shards, each containing several replica nodes. When you select Cluster Migration, you will need to fill in the Node Info.
    Enter the addresses and passwords for all shard nodes in the source database cluster (IP: port: password or IP: port). For multiple nodes, list them on separate lines.
    It is recommended to enter the addresses of the replica nodes (secondary nodes) of the source database. Migrating from replica nodes can help avoid impacting the source database's business operations.
    Public Network
    When Access Type is set to Public Network, the following parameters need to be configured:
    Host address: The IP address or domain name of the source database.
    Port: The port used by the source database.
    Password: The password for the source database migration account.
    Self-Build on CVM
    When Access Type is set to Self-Build on CVM, the following parameters need to be configured.
    CVM instance: The instance ID of the CVM
    Port: The port used by the source database
    Password: The password for the source database migration account
    Direct Connect
    When Access Type is set to Direct Connect, the following parameters need to be configured.
    VPC Direct Connect Gateway: Only the VPC Direct Connect Gateway is supported for Direct Connect. Confirm the network type associated with the gateway.
    VPC: Select the VPC and subnet.
    Host address: The IP address of the source database
    Port: The port used by the source database
    Password: The password for the source database migration account
    VPN Access
    When Access Type is set to VPN Access, the following parameters need to be configured.
    VPN gateway: VPN gateway. Select the VPN gateway instance that connects to the source database network.
    VPC: Select the VPC and subnet.
    Host address: The IP address of the source database
    Port: The port used by the source database
    Password: The password for the source database migration account
    Database
    When Access Type is set to Database, the following parameters need to be configured.
    Cross-account/Intra-account
    Same account: The source and target database instances belong to the same Tencent Cloud root account.
    Cross-Account: The source and target database instances belong to different Tencent Cloud root accounts. For detailed instructions on cross-account operations, see Cloud Database Migration Across Accounts.
    Cloud database instance: Select the instance ID of the source database.
    Password: The password for the source database migration account.
    CCN
    When Access Type is set to CCN, the following parameters need to be configured.
    When you are using CCN access, both same-account and cross-account CCN connections are supported. Due to the complexity of network configuration, see Migrating Self-built Databases to Tencent Cloud via Cloud Connect Network for detailed instructions.
    Host network environment: Select based on your actual situation.
    If the source database is a TencentDB instance, select Tencent Cloud. If the source database is an IDC self-hosted database, select Self-hosted IDC. If the source database is from another cloud provider, select the corresponding network.
    Host address: The IP address of the source database
    Port: The port used by the source database
    Password: The password for the source database migration account
    CCN instance account
    My account: The CCN resources and DTS belong to the same Tencent Cloud root account.
    Other account: The CCN resources and DTS belong to different Tencent Cloud root accounts.
    VPC CCN: The name of the CCN instance
    Access VPC: Select the VPC and subnet to be accessed. The Access VPC refers to the VPC within the CCN that is connected to the DTS migration linkage. Among all the VPCs associated with the CCN, choose one that is not the VPC to which the source database belongs.
    Access VPC region: The region of the Access VPC need to match the region of the source database selected when purchasing the task. If they do not match, DTS will automatically change the region of the source database to the Access VPC region.
    Target Database Settings
    The target database parameters are configured similarly to the source database. The following will describe only the differences.
    Configuration Parameter
    Description
    Access Type
    Select based on your scene. The configuration principles for different access types are similar to those for the source database.
    Supports migration to the cloud. When the source database access type is set to Cloud Database, the target database can be set to Public Network/Self-Built on CVM/Direct Connect/VPN Access/CCN.
    Node Type
    This parameter is displayed when any Access Type other than CDB is selected.
    Choose according to the actual situation of the target database. Supported options include Single-node migration, Redis Cluster Migration, and Proxy Cluster Migration.

    Step Ⅲ: Configuring Migration Options and Selecting Migration Objects

    On the Set migration options and migration objects tab, configure the parameters, then click Next.
    Configuration Item
    Description
    Migration Type
    The default selection is Full + Incremental Migration and cannot be modified. Full + Incremental Migration means that both the historical data from the source database before the migration and any new data written during the migration process will be migrated together.
    Migration Object
    The default selection is to migrate the entire instance and cannot be modified.

    Step Ⅳ: Validating and Starting the Task

    1. On the verification task page, the system will automatically perform a pre-check. Once the validation passes, click Start Task to immediately enable data migration. (If a scheduled execution time was set during task configuration, you can also choose Start Later here, and the task will be enabled at the scheduled time.)
    Note:
    If the validation result fails, see Processing of Pre-verification Items Not Passed.
    Validation result is failure: This indicates that the validation check did not pass, and the task is blocked. You need to fix the issue and then re-run the validation task.
    Validation result is warning: This indicates that the validation check did not fully meet the requirements. The task can continue, but it may have some impact on the business. Users need to evaluate whether to ignore the warnings or fix the issues before proceeding.
    2. Return to the data migration task list, where the task will enter the Preparing status. After 1 to 2 minutes, the data migration task will officially start.

    Step Ⅴ: Ending the Task

    1. If you need to view task progress details, delete the task, or perform other operations, click the corresponding button in the Operation column. For more details, see Task Management.
    2. When the task status is Ready to Complete, it indicates that the incremental synchronization is finished, and you can end the task. Click Complete in the Operation column to end the migration task.
    It is recommended to end the task when the data difference between the target and source databases is 0 KB and the time difference between them is 0 seconds.
    If you need to perform a business cutover after the migration is completed, see Cutover Description.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support