DTS is a data transmission service that integrates such features as data migration, sync, and subscription. It helps you migrate your databases to the cloud without interrupting your business and build a high-availability database disaster recovery architecture through real-time sync channels. Its data subscription feature meets your requirements for commercial data mining and async business decoupling.
DTS for Redis currently supports the data migration feature for you to migrate data to TencentDB in a non-stop manner at a time. In addition, in its full + incremental data migration mode, historical data in the source database written before migration and incremental data written during migration can be migrated together.
Use Cases
DTS for Redis data migration supports source and target databases in the following deployment modes:
Note:
To try out the Redis to TencentDB for Redis/KeeWiDB migration feature of NewDTS, submit a ticket for application.
Source
Target
Description
Self-built Redis database in IDC and CVM
TencentDB for Redis and TencentDB for KeeWiDB
-
Third-party Redis database
TencentDB for Redis and TencentDB for KeeWiDB
The third-party cloud vendor must grant the SYNC or PSYNC command permission.
TencentDB for Redis
TencentDB for Redis and TencentDB for KeeWiDB
Data migration, version upgrade, and cross-region migration are supported between database instances under the same Tencent Cloud account.
Redis to TencentDB for Redis migration is supported for source Redis 2.8, 3.0, 3.2, 4.0, 5.0, and 6.0. To use it for v6.0, submit a ticket for application. We recommend that you migrate from an earlier version to a later version; otherwise, compatibility problems may occur.
Redis to TencentDB for KeeWiDB migration is supported for source Redis 4.0. To migrate to other versions, submit a ticket for application.
Standard architecture and cluster architecture instances can be migrated to each other; however, such heterogeneous migration may has compatibility problems.
Supported architectures include single-node, Redis cluster, Codis, and twemproxy.
Migration permission requirements: To migrate data through DTS, the source instance must support SYNC or PSYNC commands.
Supported networks
DTS supports data migration based on the public network, CVM instances, Direct Connect gateways, VPN, and CCN.
Public Network: The source database can be accessed through a public IP.
Self-Build on CVM: The source database is deployed in a CVM instance.
Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
VPN Access: The source database can be interconnected with VPCs through VPN Connections.
CCN: The source database can be interconnected with VPCs through CCN.
Limitations
To ensure the migration efficiency, cross-region migration of CVM-based self-built instances is not supported.
To migrate instances over the public network, make sure that the source instance can be accessed over the public network.
Only instances that are running normally can be migrated, while instances with no password initialized or with ongoing tasks cannot.
The target instance must be empty. During the migration process, the target instance will be read-only and cannot be written to.
After the successfully migrated data is verified by your business, you can disconnect the source instance and connect to the target instance. This document describes how to migrate data from Redis to TencentDB for Redis with DTS.
Environment Requirements
System check
Note:
The DTS system will check the following environment requirements before starting a migration task and report an error if a requirement is not met. You can also check them in advance. For more information on error handling, see Check Item Overview.
Type
Environment Requirements
Requirements for the source database
The source and target databases can be connected.
The number of databases in the source database must be less than or equal to that in the target database.
Redis to TencentDB for Redis migration is supported for source Redis 2.2.6 or later. Redis to TencentDB for KeeWiDB migration is supported for source Redis 4.0; to migrate to other versions, submit a ticket for application.
The source database must be a replica node; otherwise, an alarm will be triggered.
Requirements for the target database
The target database version must be later than or equal to the source instance version; otherwise, an alarm will be triggered for compatibility problems during verification.
The target database must have the latest proxy.
For migration from Redis to TencentDB for Redis or TencentDB for KeeWiDB, the space of the target database must be at least 1.5 or 2 times the volume of the data to be migrated in the source database, respectively.
The target database must be empty.
Manual check
You should check and make sure that the following items are passed before the migration; otherwise, the migration may fail.
Big keys in the source database
Before the migration, check whether there are big keys in the source database. They may cause the buffer (client-output-buffer-limit) to overflow during the migration, leading to a migration failure.
For TencentDB databases, you can use the performance optimization feature of TencentDB for DBbrain to quickly analyze big keys. For more information, see Memory Analysis.
For non-TencentDB databases, use RDBTools to analyze big keys in Redis.
Evaluate large keys for splitting or cleaning. If you need to retain them, set the source buffer size (client-output-buffer-limit) to infinite.
config set client-output-buffer-limit 'slave 0 0 0'
Limit on the number of TCP connections in the source Linux kernel
If the number of concurrent business requests is high, check the limit on the number of connections in the Linux kernel before the migration. If this value is exceeded, the Linux server will actively disconnect from DTS.
Access permission of the source RDB file directory
Before the migration, check and make sure that the directory where RDB files are stored in the source database is readable; otherwise, the migration will fail.
If the RDB file directory is not readable, run the following command in the source database to set "diskless replication". Then, RDB files will be directly sent to DTS for storage, with no need to be stored in the source database first and then sent.
config set repl-diskless-sync yes
Command compatibility (for migration from standard architecture to cluster architecture)
The most challenging problem in migrating data from Standard Edition to Memory Edition (Cluster Architecture) is the command compatibility with usage specifications of Memory Edition (Cluster Architecture).
Multi-key operations
TencentDB for Redis Memory Edition (Cluster Architecture) only supports cross-slot multi-key access for mget, mset, del, and exists commands. In the source database, keys that need to engage in multi-key computing can be aggregated into the same slot through a hash tag. For more information on how to use hash tags, see Redis cluster specification.
Transactional operations
Memory Edition (Cluster Architecture) supports transactions, but cross-slot access to keys in transactions is not supported.
1.1. Log in to the DTS console, select Data Migration on the left sidebar, and click Create Migration Task on the right.
1.2. On the Create Migration Task page, select the types and regions of the source and target databases and click Buy Now.
Configuration Item
Description
Billing Mode
Only pay-as-you-go is supported.
Source Instance Type
Select the source database type, which cannot be changed after purchase. Here, select Redis.
Source Instance Region
Select the source database region. If the source database is a self-built one, select a region nearest to it.
Target Instance Type
Select the target database type, which cannot be changed after purchase. Here, select Redis.
Target Instance Region
Select the target database region.
Migration Method
Select Migrate to the cloud.
Specification
Currently, only Xlarge is supported.
Quantity
You can purchase up to 10 migration tasks at a time.
2. Set source and target databases
Configure the Source Database Settings and Target Database Settings and click Test Connectivity. After the test is passed, click Save to proceed to the next step.
Category
Configuration Item
Description
Task Configuration
Task Name
Set a task name that is easy to identify.
Running Mode
Select Immediate execution or Scheduled execution.
To modify the scheduled task, you must click Scheduled start again after the verification is passed, so as to make the task start at the specified time.
If the specified time has passed, the task will be started immediately. You can also click Immediate start to start the task immediately.
Source Database Settings
Source Database Type
The source database type selected during purchase, which cannot be changed.
Region
The region selected during purchase, which cannot be changed.
Access Type
For a third-party cloud database, you can select Public Network generally or select VPN Access, Direct Connect, or CCN based on your actual network conditions.
In this scenario, select Public Network. For the preparations for different access types, see Overview
Public Network: The source database can be accessed through a public IP.
Self-Build on CVM: The source database is deployed in a CVM instance.
Direct Connect: The source database can be interconnected with VPCs through Direct Connect.
VPN Access: The source database can be interconnected with VPCs through VPN Connections.
Database: The source database is a TencentDB instance.
CCN: The source database can be interconnected with VPCs through CCN.
Node Type
Single-Node Migration and Cluster Migration are supported. Cluster Migration is used as an example here. Currently, there are no limits on the number of shards and replicas in migration from Cluster Edition Redis to Cluster Edition Redis.
Node Info
Enter the addresses and passwords (IP:port:password or IP:port) of all shards of the source database cluster and separate the information of different nodes with line breaks. We strongly recommend you migrate data from a replica node of the source database to avoid any impact on business access to the source database.
Target Database Settings
Target Database Type
The target database type selected during purchase, which cannot be changed.
Region
The target database region selected during purchase, which cannot be changed.
Access Type
Select a type based on your scenario. In this scenario, select Database.
Database Instance
Select the instance ID of the target database.
Password
Target database password.
3. Verify and start the task
On the task verification page, verify the task. After the verification is passed, click Start Task.
Failed: It indicates that a check item fails and the task is blocked. You need to fix the problem and run the verification task again.
Alarm: It indicates that a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. You need to assess whether to ignore the alarm or fix the problem and continue the task based on the alarm message.
Return to the data migration task list, and you can see that the task has entered the Preparing status. After 1–2 minutes, the data migration task will be started. Then, full + incremental data will be migrated, that is, historical data in the source database written before migration and incremental data written during migration will be migrated together.
4. Complete the migration task
(1) (Optional) If you want to view, delete, or perform other operations on a task, click the task and select the target operation in the Operation column. For more information, see Viewing Task.
(2) If the keys of the source and target databases are the same, click Done in the Operation column to stop the data migration task.
(3) After the migration task status becomes Task successful, verify the data in the target database. If the verification is passed, you can formally cut over the business. For more information, see Cutover Description.
Event Alarming and Metric Monitoring
(1) DTS can automatically report event alarms triggered upon migration interruption to keep you informed of any exceptions. For detailed directions, see Configuring Alarm Policy for Data Migration.
(2) DTS allows you to view the monitoring data of various metrics during migration to understand the performance metrics of the system. For more information, see Viewing Monitoring Metric.
Yes
No
Was this page helpful?