Data Transfer Service
- Release Notes and Announcements
- Announcements
- Product Introduction
- Specification Description
- Preparations
- Network Preparation
- Interworking Between local IDC and Tencent Cloud
- Adding DTS IP Addresses to the Allowlist of the Corresponding Databases
- DTS Service Permission Preparation
- Data Migration
- Migration to MySQL Series
- Migrating to MySQL
- Migrating to MariaDB
- Migrating to TDSQL-C for MySQL
- Migrating to TDSQL MySQL
- Migrating to TDSQL TDStore
- Advanced Operations for MySQL Series Migration
- Processing for MySQL Series Pre-Check Failure
- Migrating to PostgreSQL
- Processing for PostgreSQL Series Pre-Check Failure
- Migrating to MongoDB
- Migration from MongoDB to MongoDB
- Fix for Verification Failure
- MongoDB Data Consistency Verification
- Migrating to SQL Server
- Migration from SQL Server to TencentDB for SQL Server
- Pre-Validation Failure Handling
- Migrating to TencentDB for Redis
- Migration from Redis to Redis
- Fix for Verification Failure
- Migrating to KeeWiDB
- Migration from Redis to KeeWiDB
- Task Management
- Data Sync
- Sync to MySQL series
- Sync to MySQL
- Sync to MariaDB
- Sync to TDSQL-C for MySQL
- Sync to TDSQL for MySQL
- Sync from TDSQL for MySQL to TDSQL for MySQL
- Sync to TDSQL TDStore
- MySQL Series Advanced Sync Operations
- MySQL series Pre Validation Failed Processing
- Synchronize to PostgreSQL
- PostgreSQL Pre Validation does not Pass Processing
- Synchronize to Kafka
- MySQL/MariaDB/Percona/TDSQL-C MySQL Sync to Kafka
- TDSQL MySQL Sync to Kafka
- Task Management
- Data Subscription (Kafka Edition)
- MySQL series Data Subscription
- Consuming MySQL Data
- Advanced Subscription Operations
- Fix for Verification Failure
- Data Subscription for TDSQL PostgreSQL
- Fix for Verification Failure
- MongoDB Data Subscription
- Fix for Verification Failure
- Task Management
- Consumption Management
- Fix for Verification Failure
- Monitoring and Alarms
- Configuring Metric Alarms and Event Alarms via the Console
- Practical Tutorial
- FAQs
- Error Handling
- API Documentation
- Making API Requests
- (NewDTS) Data Migration APIs
- Data Sync APIs
- Data Consistency Check APIs
- (NewDTS) Data Subscription APIs
- DTS API 2018-03-30
- Data Subscription APIs
- Making API Requests
- DTS APIs
- Service Agreement
Data Consumption Exception
Last updated: 2024-07-08 15:45:26
Issue
In data subscription scenarios, if you use your own consumer to consume data, you may encounter the following exceptions:
1. Data cannot be consumed.
2. The consumed data is either lost or duplicated.
3. The consumer delay keeps increasing.
Troubleshooting
1. Data cannot be consumed
If your own consumer fails to consume data, use the demo provided by DTS for consumption test first.
If the demo can normally consume data, you need to check your own consumer.
If the demo also cannot consume data, further troubleshoot as follows:
Check the network environment of the consumer. The consumer must be in the Tencent Cloud private network and in the same region as where the DTS data subscription task is.
Check whether the demo starting parameters are correct, especially the consumer group password.
Check whether the demo version is correct. The required demo version varies by source database type and data format.
Check the number of unconsumed messages on the consumer group management page in the console to see if the subscription task has written data to Kafka.
2. The consumed data is either lost or duplicated
When a data subscription task is restarted, data duplication may occur in the producer, causing the duplication of the consumed data. This scenario rarely occurs. Data duplication or data loss in other scenarios are not supposed to occur.
Generally, data duplication or data loss is caused by exceptions in your own consumer. To troubleshoot, reproduce the problem first. You can choose one of the following two methods to reproduce the problem:
In the console, change the Kafka consumption offset back to the previous offset and consume data again.
Create a consumer group and use it to consume data again. Consumption in different consumer groups does not affect each other.
If the problem can be reproduced, submit a ticket to handle it; otherwise, check whether your own consumer is abnormal.
3. The consumer delay keeps increasing
1. The commit logic of the consumer has been modified.
If the consumer only consumes data but doesn't commit the consumption offset, the offset in Kafka won't be updated. The default commit logic of the DTS demo is that, every time a checkpoint message is consumed, the consumption offset will be committed. The subscription service writes a checkpoint message about every 10 seconds. If you modify the commit rule, the consumer delay may keep increasing. To solve this problem, check the commit rule of your consumer first.
2. The consumption efficiency is too low.
The consumption efficiency may be affected by the network condition, the processing efficiency of the consumer, concurrent consumption, or consumption in multiple partitions. You can create a consumer group and compare the consumption efficiency of the DTS demo with that of your own consumer to find the cause of the low efficiency. You can also check the network condition to improve the data processing speed, or increase the number of consumers to implement concurrent consumption for topics in multiple partitions.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No