- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to TencentDB for MongoDB
- Processing for MongoDB Series Pre-Check Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Set Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- DTS-DBbridge Privatization Deployment
- Best Practices
- Local database synchronization to cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Data from Self-Built MySQL to TencentDB for MySQL Through CCN
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- DescribeMigrationDetail
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- CompleteMigrateJob
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- SDK Documentation
- Service Agreement
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to TencentDB for MongoDB
- Processing for MongoDB Series Pre-Check Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Set Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- DTS-DBbridge Privatization Deployment
- Best Practices
- Local database synchronization to cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Data from Self-Built MySQL to TencentDB for MySQL Through CCN
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- DescribeMigrationDetail
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- CompleteMigrateJob
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- SDK Documentation
- Service Agreement
DTS allows you to sync the full and incremental data of the source database to Ckafka, so that you can quickly obtain business change data and use it. This document describes how to use DTS to sync data from TDSQL for MySQL to Ckafka.
Currently, TDSQL for MySQL is the only supported source database type.
Prerequisites
The source and target databases must meet the requirements for the sync feature and version as instructed in Databases Supported by Data Sync.
Source database permissions required for the sync task account:
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT, REPLICATION SLAVE, SELECT ON *.* TO 'migration account'@'%' IDENTIFIED BY 'migration password';FLUSH PRIVILEGES;
You need to modify the message retention period and message size limit in target Ckafka.
We recommend that you set the message retention period to 3 days. The data beyond the retention period will be cleared, so you need to consume data in time within the set period. The upper limit for message size refers to the maximum size of a single message that Ckafka can receive. You must set it to be greater than the maximum size of a single row of data in the source database table so that data can be normally delivered to CKafka.
Directions
1. Log in to the data sync task purchase page, select appropriate configuration items, and click Buy Now.
Parameter | Description |
Billing Mode | Monthly subscription and pay-as-you-go billing modes are supported. |
Source Instance Type | Select TDSQL for MySQL, which cannot be changed after purchase. |
Source Instance Region | Select the source instance region, which cannot be changed after purchase. |
Target Instance Type | Select Kafka, which cannot be changed after purchase. |
Target Instance Region | Select the target instance region, which cannot be changed after purchase. |
Specification | Select a specification based on your business needs. The higher the specification, the higher the performance. For more information, see Billing Overview. |
2. After making the purchase, return to the data sync task list to view the task you just created.Then, click Configure in the Operation column to enter the Configure Sync Task page.
3. On the Configure Sync Task page, configure Instance ID, Account, and Password for the source instance, configure Instance ID for the target instance, test connectivity, and click Next.
Setting Items | Parameter | Description |
Task Configuration | Task Name | DTS will automatically generate a task name, which is customizable. |
| Running Mode | Immediate execution and scheduled execution are supported. |
Source Instance Settings | Source Instance Type | The source database type selected during purchase, which cannot be changed. |
| Source Instance Region | The source instance region selected during purchase, which cannot be changed. |
| Access Type | Select a type based on your scenario. In this scenario, you can only select Database. |
| Account/Password | Account/Password: Enter the source database account and password. |
Target Instance Settings | Target Instance Type | The target instance type selected during purchase, which cannot be changed. |
| Target Instance Region | The target instance region selected during purchase, which cannot be changed. |
| Access Type | Select a type based on your scenario. In this scenario, select CKafka instance. |
| Instance ID | Select the instance ID of the target instance. |
4. On the Set sync options and objects page, set the following items: Data Initialization Option, Policy for Syncing Data to Kafka, Data Sync Option, and Sync Object Option. Then click Save and Go Next.
Deliver to custom topic
Deliver to a single topic
Setting Items | Parameter | Description |
Data Initialization Option | Initialization Type | Structure initialization: Table structures in the source instance will be initialized into the target instance before the sync task runs. Full data initialization: Data in the source instance will be initialized into the target instance before the sync task runs. If you select Full data initialization only, you need to create the table structures in the target database in advance. Both options are selected by default, and you can deselect them as needed. |
| Format of Data Delivered to Kafka | Avro adopts the binary format with a higher consumption efficiency, while JSON adopts the easier-to-use lightweight text format. |
Policy for Syncing Data to Kafka | Topic Sync Policy | Deliver to custom topic: Customize the topic name for delivery. After that, the target Kafka will automatically create a topic with the custom name. The synced data is randomly delivered to different partitions under the topic. If the target Kafka fails to create the topic, the task will report an error. Deliver to a single topic: Select an existing topic on the target side, and then deliver data based on multiple partitioning policies. Data can be delivered to a single partition of the specified topic, or delivered to different partitions by table name or by table name + primary key. |
| Rules for delivering to custom topic | If you add multiple rules, the database and table rules are matched one by one from top to bottom. If no rules are matched, data will be delivered to the topic corresponding to the last rule. If multiple rules are matched, data will be delivered to the topics corresponding to all the matched rules. Example 1: There are tables named "Student" and "Teacher" in a database named "Users" on database instance X. If you need to deliver the data in the "Users" database to a topic named "Topic_A". The rules are configured as follows: Enter Topic_A for Topic Name, ^Users$ for Database Name Match, and .* for Table Name Match.Enter Topic_default for Topic Name, Databases that don't match the above rules for Database Name Match, and Tables that don't match the above rules for Table Name Match.Example 1: There are tables named "Student" and "Teacher" in a database named "Users" on database instance X. If you need to deliver the data in the "Student" table and "Teacher" tables to topics named "Topic_A" and "Topic_default" respectively. The rules are configured as follows: Enter Topic_A for Topic Name, ^Users$ for Database Name Match, and ^Student$ for Table Name Match.Enter Topic_default for Topic Name, Databases that don't match the above rules for Database Name Match, and Tables that don't match the above rules for Table Name Match. |
| Rules for delivering to a single topic | After selecting a specified topic, the system will perform partitioning based on the specified policy as follows. Deliver all data to partition 0: Deliver all the synced data of the source database to the first partition. By table name: Partition the synced data from the source database by table name. After setting, the data with the same table name will be written into the same partition. By table name + primary key: Partition the synced data from the source database by table name and primary key. This policy is suitable for frequently accessed data. After settings, frequently accessed data is distributed from tables to different partitions by table name and primary key, so as to improve the concurrent consumption efficiency. |
| Topic for DDL Storage | (Optional) If you need to deliver the DDL operation of the source database to the specified topic separately, you can select the settings here.
After setting, it will be delivered to Partition 0 of the selected topic by default; if not set, it will be delivered based on the topic rules selected above. |
Data Sync Option
Setting Items | Parameter | Description |
Data Sync Option | SQL Type | The following operations are supported: INSERT, DELETE, UPDATE, and DDL. |
Sync Object Option | Database and Table Objects of Source Instance | Only the database/table objects can be synced. |
5. On the task verification page, complete the verification. After all check items are passed, click Start Task.
If the verification fails, fix the problem as instructed in Check Item Overview and initiate the verification again.
Failed: It indicates that a check item fails and the task is blocked. You need to fix the problem and run the verification task again.
Alarm: It indicates that a check item doesn't completely meet the requirements, and the task can be continued, but the business will be affected. You need to assess whether to ignore the alarm or fix the problem and continue the task based on the alarm message.
6. Return to the data sync task list, and you can see that the task has entered the Running status.
Note
You can click More > Stop in the Operation column to stop a sync task. Before doing so, ensure that data sync has been completed.
7. (Optional) you can click a task name to enter the task details page and view the task initialization status and monitoring data.
Subsequent Operations
After the data is synced to the target Kafka, the data can be consumed. We provide you with a c
Yes
No
Was this page helpful?