- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to TDSQL TDStore
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Instance Permission Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL Kernel Check
- DDL Ring Linkage Synchronization Detection for Same Database and Table Objects
- DDL Synchronization Conflict Detection for the Same Target and Same Database and Table Object
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- PostgreSQL Physical Migration
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to MongoDB
- Fix for Verification Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Collection Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit (MySQL)
- Speed Limit (Redis)
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- Sync to TDSQL TDStore
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- Data Subscription for TDSQL PostgreSQL
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- Practical Tutorial
- Synchronizing Local Database to the Cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Self-Built Databases to Tencent Cloud Databases via CCN
- Best Practices for DTS Performance Tuning
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- CompleteMigrateJob
- ContinueMigrateJob
- DescribeMigrateDBInstances
- DescribeMigrationDetail
- DestroyMigrateJob
- IsolateMigrateJob
- ModifyMigrateJobSpec
- ModifyMigrateName
- ModifyMigrateRuntimeAttribute
- PauseMigrateJob
- RecoverMigrateJob
- ResumeMigrateJob
- StopMigrateJob
- ModifyMigrateRateLimit
- Data Sync APIs
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- DescribeCheckSyncJobResult
- SkipSyncCheckItem
- StartSyncJob
- IsolateSyncJob
- RecoverSyncJob
- PauseSyncJob
- ContinueSyncJob
- ResumeSyncJob
- StopSyncJob
- DestroySyncJob
- ResizeSyncJob
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- ModifySyncRateLimit
- Data Consistency Check APIs
- (NewDTS) Data Subscription APIs
- CreateSubscribe
- ConfigureSubscribeJob
- CreateSubscribeCheckJob
- DescribeSubscribeCheckJob
- StartSubscribe
- ModifySubscribeObjects
- DescribeSubscribeDetail
- DescribeSubscribeJobs
- ResumeSubscribe
- CreateConsumerGroup
- DescribeConsumerGroups
- ModifyConsumerGroupDescription
- ModifyConsumerGroupPassword
- DescribeOffsetByTime
- ResetConsumerGroupOffset
- DeleteConsumerGroup
- ResetSubscribe
- ModifySubscribeAutoRenewFlag
- ModifySubscribeName
- DescribeSubscribeReturnable
- IsolateSubscribe
- DestroyIsolatedSubscribe
- Data Types
- Error Codes
- DTS API 2018-03-30
- Service Agreement
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Preparations
- Business Evaluation
- Network Preparation
- DTS Service Permission Preparation
- Database and Permission Preparation
- Configuring Binlog in Self-Built MySQL
- Data Migration
- Databases Supported by Data Migration
- Cross-Account TencentDB Instance Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to TDSQL TDStore
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Instance Permission Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL Kernel Check
- DDL Ring Linkage Synchronization Detection for Same Database and Table Objects
- DDL Synchronization Conflict Detection for the Same Target and Same Database and Table Object
- Offline Migration of MySQL Data
- Migrating to PostgreSQL
- Migration from PostgreSQL to TencentDB for PostgreSQL
- Migration from TDSQL-C PostgreSQL to TencentDB for PostgreSQL
- PostgreSQL Physical Migration
- Processing for PostgreSQL Series Pre-Check Failure
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Multi Task Conflict Detection
- Long Transaction Check
- Migrating to MongoDB
- Migration from MongoDB to MongoDB
- Fix for Verification Failure
- MongoDB Connection Check
- Database/Table Content Conflict Check
- Source Database Node Role Check
- Oplog Check
- Source/Target Database Account Permission Check
- Database Version Check
- Database Capacity Check
- Target Database Load Check
- ShardKey Check
- Source Database Balancer Check
- Time Series Collection Verification
- Compression Algorithm Verification
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migrating to TencentDB for Redis
- Migrating to KeeWiDB
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Start Task
- Retrying Task (MySQL)
- Retrying Task (Redis)
- Speed Limit (MySQL)
- Speed Limit (Redis)
- Complete Task
- Terminating Task
- Deleting Task
- Pausing Task
- Resuming Task
- Changing Configuration
- Terminating Task
- Deisolating Task
- Deleting Task
- Managing Tag
- Creating a Similar Task
- Viewing Log
- Data Sync
- Databases Supported by Data Sync
- Cross-Account TencentDB Instance Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- Sync to TDSQL TDStore
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Database Connection Check
- Peripheral Check
- Version Check
- Source Instance Permission Check
- Partial Database Parameter Check
- Target Database Content Conflict Check
- Target Database Space Check
- Binlog Parameter Check
- Foreign Key Dependency Check
- View Check
- Advanced Object Check
- Warning Item Check
- TDSQL Proxy Check
- TDSQL kernel check
- DLL Ring Sync Check for Single Database/Table Object
- DLL Sync Conflict Check for Single Database/Table Object in the Same Target
- Synchronize to PostgreSQL
- Synchronization from PostgreSQL to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Database Connection Check
- Object Dependency Check
- Version Check
- Source or Target Instance Permission Check
- Incremental Migration Pre Check
- Structural Compatibility Check
- Structural Conflict Check
- Account Conflict Check
- Parameter Configuration Conflict Check
- Plugin Compatibility Check
- Primary Key Dependency Check
- Check the Key Parameters of the Instance
- Multi Task Conflict Detection
- Synchronize to Kafka
- Task Management
- Task Status Description
- Viewing Task
- Configuring Task
- Verifying Task
- Starting Task
- Retrying Task (MySQL)
- Speed Limit
- Stopping Task
- Modify Sync configuration
- Pausing Task
- Resuming Task
- Resetting Task
- Deleting Task
- Changing Configuration
- Pay-as-You-Go to Monthly Subscription
- Terminating/Returning Instance
- Renewing Task
- Deisolating Task
- Deleting Instance
- Creating a Similar Task
- Viewing Log
- Deleting Task
- Data Subscription (Kafka Edition)
- Databases Supported by Data Subscription
- MySQL series Data Subscription
- Creating MySQL or TDSQL for MySQL Data Subscription
- Creating TDSQL for MySQL Data Subscription
- Creating MariaDB/Percona Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- Data Subscription for TDSQL PostgreSQL
- MongoDB Data Subscription
- Task Management
- Consumption Management
- Fix for Verification Failure
- Cutover Description
- Monitoring and Alarms
- Ops Management
- Practical Tutorial
- Synchronizing Local Database to the Cloud
- Creating Two-Way Sync Data Structure
- Creating Many-to-One Sync Data Structure
- Creating Multi-Site Active-Active IDC Architecture
- Selecting Data Sync Conflict Resolution Policy
- Using CLB as Proxy for Cross-Account Database Migration
- Migrating Self-Built Databases to Tencent Cloud Databases via CCN
- Best Practices for DTS Performance Tuning
- FAQs
- Error Handling
- API Documentation
- History
- Introduction
- Introduction
- API Category
- Making API Requests
- (NewDTS) Data Migration APIs
- DescribeMigrationJobs
- CreateMigrationService
- ModifyMigrationJob
- CreateMigrateCheckJob
- DescribeMigrationCheckJob
- SkipCheckItem
- StartMigrateJob
- CompleteMigrateJob
- ContinueMigrateJob
- DescribeMigrateDBInstances
- DescribeMigrationDetail
- DestroyMigrateJob
- IsolateMigrateJob
- ModifyMigrateJobSpec
- ModifyMigrateName
- ModifyMigrateRuntimeAttribute
- PauseMigrateJob
- RecoverMigrateJob
- ResumeMigrateJob
- StopMigrateJob
- ModifyMigrateRateLimit
- Data Sync APIs
- DescribeSyncJobs
- CreateSyncJob
- ConfigureSyncJob
- CreateCheckSyncJob
- DescribeCheckSyncJobResult
- SkipSyncCheckItem
- StartSyncJob
- IsolateSyncJob
- RecoverSyncJob
- PauseSyncJob
- ContinueSyncJob
- ResumeSyncJob
- StopSyncJob
- DestroySyncJob
- ResizeSyncJob
- ModifySyncJobConfig
- CreateModifyCheckSyncJob
- DescribeModifyCheckSyncJobResult
- StartModifySyncJob
- ModifySyncRateLimit
- Data Consistency Check APIs
- (NewDTS) Data Subscription APIs
- CreateSubscribe
- ConfigureSubscribeJob
- CreateSubscribeCheckJob
- DescribeSubscribeCheckJob
- StartSubscribe
- ModifySubscribeObjects
- DescribeSubscribeDetail
- DescribeSubscribeJobs
- ResumeSubscribe
- CreateConsumerGroup
- DescribeConsumerGroups
- ModifyConsumerGroupDescription
- ModifyConsumerGroupPassword
- DescribeOffsetByTime
- ResetConsumerGroupOffset
- DeleteConsumerGroup
- ResetSubscribe
- ModifySubscribeAutoRenewFlag
- ModifySubscribeName
- DescribeSubscribeReturnable
- IsolateSubscribe
- DestroyIsolatedSubscribe
- Data Types
- Error Codes
- DTS API 2018-03-30
- Service Agreement
Overview
In data subscription (Kafka Edition, where the current Kafka Server version is 2.6.0), subscribed data in Protobuf format can be consumed by using a Flink client (only the DataStream API type). This document provides a demo for data consumption with consumer-demo-tdsql-pb-flink for you to quickly test the process of data consumption and understand the method of data parsing.
Prerequisites
1. You have created a data consumption task as instructed in Creating Data Subscription Task.
2. You have created a consumer group as instructed in Creating Consumer Group.
3. You have installed Flink, and it can execute tasks normally.
Note
The demo only prints out the consumed data and contains no instructions. You need to write your own data processing logic based on the demo.
Currently, data subscription to Kafka for consumption can only be implemented over the Tencent Cloud private network but not the public network. In addition, the subscribed database instance and the data consumer must be in the same region.
In scenarios where only the specified databases and tables (part of the source instance objects) are subscribed and single-partition Kafka topics are used, only the data of the subscribed objects will be written to Kafka topics after DTS parses the incremental data. The data of non-subscription objects will be converted into empty transactions and then written to Kafka topics. Therefore, there are empty transactions during message consumption. The BEGIN and COMMIT messages in the empty transactions contain the GTID information, which can ensure the GTID continuity and integrity.
To ensure that data can be rewritten from where the task is paused, DTS adopts the checkpoint mechanism for data subscription. Specifically, when messages are written to Kafka topics, a checkpoint message is inserted every 10 seconds to mark the data sync offset. When the task is resumed after being interrupted, data can be rewritten from the checkpointed offeset. The consumer commits a consumption offset every time it encounters a checkpoint message so that the consumption offset can be updated timely.
Downloading Consumption Demo
For the description of the logic and key parameters in the demo, see Protobuf Demo Description (Flink).
When configuring the subscription task, you can only select Protobuf as the format of the subscribed data for TDSQL for MySQL. As Protobuf adopts the binary format, the consumption efficiency is high. The following demo already contains the Protobuf protocol file, so you don't need to download it separately. If you download it on your own, use Protobuf 3.X for code generation to ensure that data structures are compatible.
Demo Language | Protobuf (TDSQL for MySQL) |
Java |
Directions for the Java Flink Demo
Compiling environment: Maven and JDK8. You can choose a desired package management tool. The following takes Maven as an example.
Runtime environment: Tencent Cloud CVM (which can access the private network address of the Kafka server only if it is in the same region as the subscribed instance). Install JRE8.
Directions:
1. Download the consumer-demo-tdsql-pb-flink.zip file and unzip it.
2. Access the directory of the unzipped file. The
pom.xml file has been placed under the directory for your convenience. You need to modify the Flink cluster version to the same as that specified in the pom.xml dependency.3. The value of
${flink.version} in the code below must be the same as the Flink cluster version.<dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_${scala.binary.version}</artifactId><version>${flink.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-connector-kafka_${scala.binary.version}</artifactId><version>${flink.version}</version><exclusions><exclusion><groupId>org.apache.kafka</groupId><artifactId>kafka-clients</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients_${scala.binary.version}</artifactId><version>${flink.version}</version></dependency>
4. Go to the directory where the
pom.xml file is located and package it with Maven or IEDA.
Package with Maven by running mvn clean package.5. If the Flink client type is DataStream API, use Flink client commands to submit the job to the Flink cluster and start consumption.
./bin/flink run consumer-demo-tdsql-pb-flink.jar --brokers xxx --topic xxx --group xxx --user xxx --password xxx --trans2sqlbrokers is the private network access address for data subscription to Kafka, and topic is the subscription topic, which can be viewed on the Subscription details page as instructed in Viewing Subscription Details.group, user, and password are the name, account, and password of the consumer group, which can be viewed on the Consumption Management page as instructed in Managing Consumer Group.trans2sql indicates whether to enable conversion to SQL statement. In Java code, if this parameter is carried, the conversion will be enabled.6. Execute DML statements in the source database.
CREATE TABLE `flink_test` (`id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,`parent_id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,`user_id` bigint NOT NULL,`type` int NOT NULL COMMENT '1: Expenditure 2: Income',`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),PRIMARY KEY (`id`) USING BTREE) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ROW_FORMAT=DYNAMIC
7. Observe the consumption performed by the previously submitted job.
View the specific task logs on task managers.
View the specific Stdout information on task managers.
Yes
No
Was this page helpful?