Step 3. Migrating Data to Cloud

Recent Pages

Step 3. Migrating Data to Cloud

Last updated: 2024-08-07 14:25:27

Overview
This document mainly introduces how to use the dual-producer dual-consumer scheme to switch the services of a self-built RabbitMQ cluster to TDMQ for RabbitMQ.
Scheme: Dual-Producer Dual-Consumer Mode
Prerequisites
1. Purchased a RabbitMQ Instance on Cloud
2. Migrated the metadata of the self-built RabbitMQ cluster to Tencent Cloud RabbitMQ.
Directions
1. Switch the access information for some nodes in the consumer cluster and transfer these consumers to the new RabbitMQ cluster. These consumers will consume messages from the new RabbitMQ cluster, while the remaining consumers will continue to consume messages from the old RabbitMQ cluster.
2. Switch the access information for some nodes in the producer cluster and transfer these producers to the new RabbitMQ. These producers will send messages to the new RabbitMQ cluster, while the remaining producers will continue to send messages to the old RabbitMQ cluster. To prevent duplicate or lost messages, you can implement idempotent logic for message consumption in advance.
3. Transfer the remaining producers to the new RabbitMQ cluster. Then, all messages will be sent to the new RabbitMQ cluster.
 Tips 1:  You can confirm that the production traffic of the self-built RabbitMQ cluster has stopped in the community management console.
﻿
 Tips 2 : You can confirm that the heaped messages in the self-built RabbitMQ cluster are decreasing in the community management console.
﻿
4. Check if there are any unconsumed messages heaped in the old RabbitMQ cluster. After you confirm that there are no heaped or unhandled messages in the old RabbitMQ cluster, transfer the remaining consumers to the new RabbitMQ cluster to complete the migration of the entire data flow.
 Tips:  Confirm the message production and consumption in the cloud RabbitMQ cluster, and ensure there is no message heap.
﻿
 Note:
Follow the above steps strictly. If you switch producers first and then switch consumers, message loss may occur.
Before you switch the remaining consumers, ensure that all messages in the old RabbitMQ cluster have been consumed to avoid any missed consumption.
Possible Issues
Order Issue
Due to the cluster switch, the order of messages cannot be guaranteed during the switch process. There may be partial disorder during the switch.
Message Duplication
In theory, Message will not duplicate, but in extreme cases it can occur. For example, during the switch process, a consumer has consumed a message but has not sent an ACK to the server (the old RocketMQ cluster). This can cause the message to enter the retry queue, leading to duplicate consumption. Implementing idempotent logic for the messages can avoid this issue.
Consumption Delay
During the consumer switch process, the reallocation of partitions requires rebalance between queues and consumer clients, which may cause short consumption delays. No additional operations are needed in this situation, and operations will resume once the switch is completed.