tencent cloud

All product documents
TDMQ for CKafka
DocumentationTDMQ for CKafkaOperation GuideMigration to CloudStep 3. Migrating Data to CloudScheme 1: Single-Producer Dual-Consumer Migration
Scheme 1: Single-Producer Dual-Consumer Migration
Download PDF
Last updated: 2024-01-09 14:49:39
Scheme 1: Single-Producer Dual-Consumer Migration
Last updated: 2024-01-09 14:49:39
Download PDF

Overview

This document describes how to use the single-producer dual-consumer scheme to migrate data from a self-built Kafka cluster to a CKafka cluster.

Prerequisites

You have already purchased a CKafka instance.
You have already migrated the topic to CKafka.

Directions

If your requirement for message ordering is not high, you can migrate the data while it is consumed by multiple consumers in parallel.
The single-producer dual-consumer scheme is simple, clear, and easy to implement, with no data heap for smooth transition; however, it requires adding a new consumer.
The migration steps are as follows:



1. Keep the old consumer intact, start a new consumer in CKafka, and configure the bootstrap-server of the new CKafka cluster for consumption.
You need to configure the accessed network of the CKafka instance as the IP in --bootstrap-server by copying the information in the Network column in the Access Mode section on the Instance Details page in the console.
./kafka-console-consumer.sh --bootstrap-server xxx.xxx.xxx.xxx:9092 --from-beginning --new-consumer --topic topicName --consumer.config ../config/consumer.properties
2. Switch the production flow so that the producer produces data to the CKafka instance. Change the IP in the broker-list to the accessed network of the CKafka instance and topicName to the topic name in the CKafka instance:
 ./kafka-console-producer.sh --broker-list xxx.xxx.xxx.xxx:9092 --topic topicName
3. The original consumer does not need to be configured and can continue to consume the data in your self-built Kafka cluster. After such data is all consumed, the migration is completed.
Note:
The above commands are test commands. In actual business operations, just modify the broker address configured for the corresponding application and then restart the application.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon