tencent cloud

$0 14-Day TrialExperience EdgeOne for acceleration and security protection!

Feedback

TDMQ for CKafka

Scheme 1: Single-Producer Dual-Consumer Migration

Last updated: 2024-01-09 14:49:39

Overview

This document describes how to use the single-producer dual-consumer scheme to migrate data from a self-built Kafka cluster to a CKafka cluster.

Prerequisites

You have already purchased a CKafka instance.

Directions

If your requirement for message ordering is not high, you can migrate the data while it is consumed by multiple consumers in parallel.
The single-producer dual-consumer scheme is simple, clear, and easy to implement, with no data heap for smooth transition; however, it requires adding a new consumer.
The migration steps are as follows:



1. Keep the old consumer intact, start a new consumer in CKafka, and configure the bootstrap-server of the new CKafka cluster for consumption.
You need to configure the accessed network of the CKafka instance as the IP in --bootstrap-server by copying the information in the Network column in the Access Mode section on the Instance Details page in the console.
./kafka-console-consumer.sh --bootstrap-server xxx.xxx.xxx.xxx:9092 --from-beginning --new-consumer --topic topicName --consumer.config ../config/consumer.properties
2. Switch the production flow so that the producer produces data to the CKafka instance. Change the IP in the broker-list to the accessed network of the CKafka instance and topicName to the topic name in the CKafka instance:
./kafka-console-producer.sh --broker-list xxx.xxx.xxx.xxx:9092 --topic topicName
3. The original consumer does not need to be configured and can continue to consume the data in your self-built Kafka cluster. After such data is all consumed, the migration is completed.
Note:
The above commands are test commands. In actual business operations, just modify the broker address configured for the corresponding application and then restart the application.


Catalog

In The Article

Scheme 1: Single-Producer Dual-Consumer Migration

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon