tencent cloud

All product documents
TDMQ for Apache Pulsar
DocumentationTDMQ for Apache PulsarOperation GuideCluster MigrationCluster Migration Capability Description
Cluster Migration Capability Description
Download PDF
Last updated: 2025-04-01 14:19:29
Cluster Migration Capability Description
Last updated: 2025-04-01 14:19:29
Download PDF

Application Scenarios

To meet user needs in different usage scenarios, TDMQ Pulsar provides two product forms: professional cluster and virtual cluster.
Virtual clusters are at risk of instability. We stopped adding new ones in 2023. Professional clusters have stronger product capabilities and a more optimized console (for management, renewal, scale-out, etc.). To provide better service, for your virtual clusters currently in use, we provide the ability for smooth inter-cluster migration, supporting smooth migration from virtual clusters to professional clusters.

Capability Description

The migration process of cluster data is almost transparent to users, that is, the migration process is smooth (no adjustment needed for access points, no code modification required for users' services).
Migration Process
1. The system will deduct according to the specifications of the professional cluster. When starting migration, the deduction status can be seen as in processing on the order interface; during the migration process, if a rollback occurs due to a problem, the order will be automatically refunded; after migration is completed, the order status is completed and timing officially starts.
2. On the List Interface of Professional Cluster in the console, you can see the cluster information after migration and complete subsequent operations such as management, upgrade, and renewal.

Smooth Prerequisite

Smooth migration has one prerequisite: Some access point addresses are unable to smoothly migrate to the new cluster because the cluster was created earlier or the network was established in a special way.
Therefore, users need to provide information about the access point address in use to confirm the feasibility.

Operation Process

1. Initiate migration. The frontend will initiate verification of the current access point address. If there is a non-standard access point, a pop-up notification will appear.
2. Target specification selection. Based on the current concurrency status of the virtual cluster, you can select a professional cluster with the corresponding specification.
3. Access point scanning. The server will scan whether the client is using a non-standard access point;
4. Initiate upgrade. Start the migration action.

Possible Issues

1. Message duplication issue
Idempotent processing has been realized for progress synchronization through individual ack to reduce the number of duplicate messages during the migration process as much as possible. However, duplicates during the migration process may not be completely avoided. Usually, the duplication time does not exceed 1 minute. If necessary, users are required to perform idempotent processing in advance.
2. Message disorder
A potential issue that may exist in cluster migration. No solution can completely avoid the disorderly situation during the migration process. Users need to be notified in advance.
3. Monitoring data
Switching clusters may lead to inaccurate instantaneous monitoring data, which usually recovers within 1 - 2 minutes.
4. Fluctuation in generation time
Switching clusters may cause temporary fluctuations in generation time. The time taken is similar to that during the cluster upgrade process. It usually recovers within 1 minute.
5. Message trace
During the process of data synchronization, there will be messages generated for consumption progress synchronization. Users will see such messages when using message query. Querying message details may have certain impacts during the migration process. There may be situations where they cannot be viewed.
6. Migration duration
The time for the entire migration is related to the quantity of namespaces, production flow, and the quantity of stored data in message storage. For a namespace, with 1000 TPS and 100G message storage, the cluster migration can usually be completed within 1 hour; in the case of a large data volume, for example, 1T storage will probably take about 2 hours.
7. Retention time of the existing cluster
After migration, the Tencent Cloud product research and development side will wait for 1 - 3 days before cleaning up the resources on the old physical cluster. After clearance, it will not be able to roll back.
8. Message backlog issue
The synchronization of consumption progress during the migration process is performed through the user's topic. Therefore, there will be some internal messages in the user's topic. These messages will be filtered out on the server side when consumed. In business reality, these messages will not actually be consumed. For subscriptions without consumers, there will be a message backlog situation.
9. Message replication scope
During the message replication process, due to Pulsar's implementation mechanism, only messages within the TTL range of the existing cluster can be guaranteed to be replicated to the new cluster. If your message retention period is relatively large and you need to synchronize the data within the retention period, you need to adjust the TTL configuration first.

Migration Principle

Technical Solution

Adopt the Pulsar Geo Replication solution and enable cross-cluster two-way replication to achieve synchronization of data and message progress to meet the needs of cluster migration.



Migration Main Process

The main process of migration is shown as illustrated below. Go to next step upon success of each stage.

1. Console operations initiate the migration action. The platform initiates the shipment. After you pay for the order, you start building a new professional cluster.
2. Cluster metadata synchronization, namespace, topic, subscription, role, namespace policies, etc.
3. Enable cross-cluster data synchronization.
4. Switch cluster. The operation platform sends a tenant cluster switch instruction to the old cluster, actively triggering the unload of topics on the old cluster, triggering the client lookup, and returning the new cluster address information during the lookup stage.
5. Confirm that the user data has been successfully migrated, then disable cross-cluster data synchronization and clean up the old cluster resources.
6. You can see the new cluster in the console.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 available.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon