- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to TDSQL TDStore
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Instance Permission Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL Kernel Check
- DDL Ring Linkage Synchronization Detection for Same Database and Table Objects
- DDL Synchronization Conflict Detection for the Same Target and Same Database and Table Object
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- PostgreSQL Physical Migration
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to MongoDB
- Fix for Verification Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Collection Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit (MySQL)
- Speed Limit (Redis)
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- Sync to TDSQL TDStore
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- Data Subscription for TDSQL PostgreSQL
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- Practical Tutorial
- Synchronizing Local Database to the Cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Self-Built Databases to Tencent Cloud Databases via CCN
- Best Practices for DTS Performance Tuning
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- ModifyMigrationJob
- CompleteMigrateJob
- DescribeMigrationDetail
- CreateMigrationService
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- DescribeSyncJobs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- Service Agreement
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to TDSQL TDStore
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Instance Permission Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL Kernel Check
- DDL Ring Linkage Synchronization Detection for Same Database and Table Objects
- DDL Synchronization Conflict Detection for the Same Target and Same Database and Table Object
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- PostgreSQL Physical Migration
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to MongoDB
- Fix for Verification Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Collection Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit (MySQL)
- Speed Limit (Redis)
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- Sync to TDSQL TDStore
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- Data Subscription for TDSQL PostgreSQL
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- Practical Tutorial
- Synchronizing Local Database to the Cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Self-Built Databases to Tencent Cloud Databases via CCN
- Best Practices for DTS Performance Tuning
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- ModifyMigrationJob
- CompleteMigrateJob
- DescribeMigrationDetail
- CreateMigrationService
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- StopMigrateJob
- ResumeMigrateJob
- RecoverMigrateJob
- ModifyMigrateName
- ModifyMigrateJobSpec
- IsolateMigrateJob
- DestroyMigrateJob
- DescribeMigrateDBInstances
- PauseMigrateJob
- ContinueMigrateJob
- Data Sync APIs
- DescribeSyncJobs
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- SkipSyncCheckItem
- DescribeCheckSyncJobResult
- StartSyncJob
- StopSyncJob
- ResumeSyncJob
- ResizeSyncJob
- RecoverSyncJob
- IsolateSyncJob
- DestroySyncJob
- PauseSyncJob
- ContinueSyncJob
- Data Consistency Check APIs
- Data Types
- Error Codes
- DTS API 2018-03-30
- Service Agreement
This document describes how to create a data subscription task in DTS for MariaDB and Percona. MariaDB is used as an example, and the related operations also apply to Percona.
Prerequisites
You have prepared a source database to be subscribed to, and the database version meets the requirements. For more information, see Databases Supported by Data Subscription.
You have enabled the binlog in the source database.
You have created a subscription account in the source database and granted the account the following permissions: REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT for all objects.
Authorization statements are as follows:
CREATE USER 'account' IDENTIFIED BY 'password';GRANT SELECT, REPLICATION CLIENT,REPLICATION SLAVE,PROCESS ON *.* TO 'account'@'%';FLUSH PRIVILEGES;
Restrictions
Currently, the subscribed message content is retained for one day by default. The data will be cleared when the retention period ends. Therefore, you need to consume the data promptly.
The region where the data is consumed should be the same as that of the subscribed instance.
Geometry data types are not supported currently.
During the process of subscribing to a task, if you perform operations such as modifying the subscription object, the task will be restarted, which may cause duplicated data consumption on the Kafka client.
DTS transfers data based on the smallest data unit. Every time the incremental data is marked with a checkpoint, it will be considered a data unit. If the transfer of one data unit is completed before the task restart, it will not cause data duplication; if one data unit is still being transferred during the task restart, the data unit needs to be pulled again after the restart to ensure data integrity, which will lead to data duplication.
If you are more concerned about duplicate data, you can implement deduplication logic when consuming data.
Subscribable SQL Operations
Operation Type | Supported SQL Operations |
DML | INSERT, UPDATE, DELETE |
DDL | CREATE DATABASE, DROP DATABASE, CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE |
Directions
1. Log in to the DTS console, select Data Subscription on the left sidebar, and click Create Subscription.
2. On the Create Subscription page, select the corresponding configuration and click Buy Now.
Billing Mode: Monthly subscription and pay-as-you-go billing are supported.
Region: If the source database is a TencentDB database, select the region of the source database instance here. Otherwise, select the region where data is consumed. If there is no special requirement for the data consumption region, select a region closest to the source database.
Database: Select the database type.
Edition: Select Kafka Edition. You can directly consume data on a Kafka client.
Subscribed Instance Name: Edit the name of the currently subscribed instance.
3. After successful purchase, return to the data subscription list. You need to click Configure Subscription in the Operation column to configure the newly purchased subscription before you can use it.
4. On the subscription configuration page, select the source database information, click Test Connectivity, and click Next after the test is passed.
Configuration Item | Description |
Instance Type | The database type selected during purchase. We recommend that you subscribe to the primary database because the subscription service only puts a little pressure on the source database. If the source database is TencentDB for MariaDB, you can select a disaster recovery or read-only instance. |
Access Type | Select a type based on your scenario. For the preparations for different access types, see Preparations > Overview. Public Network: The source database can be accessed through a public IP. Self-Build on CVM: The source database is deployed on a CVM instance. For more information on CVM, see Cloud Virtual Machine (CVM). Direct Connect: The source database can be interconnected with VPCs through Direct Connect. For more information on Direct Connect, see Direct Connect. VPN Access: The source database can be interconnected with VPCs through VPN Connections. For more information on VPN Connections, see VPN Connections. Database: The source database is a TencentDB database. CCN: The source database can be interconnected with VPCs through CCN. For more information on CCN, see Cloud Connect Network (CCN). |
Public Network | Host Address: IP address or domain name of the source database. Port: Port used by the source database. |
Self-Build on CVM | CVM Instance: The ID of the CVM instance. Port: Port used by the source database. |
Direct Connect | VPC-Based Direct Connect Gateway: Only VPC-based direct connect gateway is supported. Confirm the network type associated with the gateway. VPC: Select a VPC and subnet associated with the VPC-based Direct Connect gateway or VPN gateway. Host Address: IP address of the source database. Port: Port used by the source database. |
VPN Access | VPN Gateway: Select a VPN gateway instance. VPC: Select a VPC and subnet associated with the VPC-based Direct Connect gateway or VPN gateway. Host Address: IP address of the source database. Port: Port used by the source database. |
Database | Instance Name: The ID of the source database instance. |
CCN | Host Address: IP address of the source database server. Port: Port used by the source database. VPC-Based CCN Instance: The name of the CCN instance. Accessed VPC: It refers to the VPC in CCN over which the subscription link is connected. You need to select a VPC other than the VPC to which the source database belongs.
For example, if the database in Guangzhou is used as the source database, select another region, such VPC-Chengdu or VPC-Shanghai, as the accessed VPC. Subnet: Name of the subnet of the selected VPC. Region of Accessed VPC: The region of the source database selected during task purchase must be the same as the region of the accessed VPC; otherwise, DTS will change the former to the latter. |
Account/Password | Account/Password: Enter the database account and password. |
Number of Kafka Partitions | Set the number of Kafka partitions. Increasing the number can improve the speed of data write and consumption. A single partition can guarantee the order of messages, while multiple partitions cannot. If you have strict requirements for the order of messages during consumption, set this value to 1. |
5. On the Subscription Type and Object page, select a subscription type and click Save.
Subscription Type: Options include Data Update, Structure Update, and Full Instance.
Data Update: Data updates of the selected objects are subscribed to, including INSERT, UPDATE, and DELETE operations.
Structure Update: creation, modification, and deletion of the structures of all objects in the instance are subscribed to.
Full Instance: Data and structure updates of all objects in the instance are subscribed to.
Kafka Partitioning Policy: Select By table name or By table name + primary key.
6. On the Pre-verification page, a pre-verification task will run for 2–3 minutes. After the pre-verification is passed, click Start to complete the data subscription task configuration.
Note
If the verification fails, fix the problem as instructed in Check Item Overview and initiate the verification again.
7. The subscription task will be initialized, which will take 3–4 minutes. After successful initialization, the task will enter the Running status.
8. Add a consumer group. Data subscription (Kafka Edition) allows you to create multiple consumer groups for multi-point consumption. For more information, see Creating Consumer Group. The consumption in data subscription (Kafka Edition) depends on the consumer groups of Kafka; therefore, you must create a consumer group first before data can be consumed.
9. After the subscription instance enters the Running status, you can start consuming data. To consume data in Kafka, you need to verify the password. For specific examples, see Consuming MySQL Data. We provide demo code in multiple programming languages and descriptions of main consumption processes and key data structures.
Yes
No
Was this page helpful?