Pulsar supports multi-AZ deployment. When you purchase a Pulsar cluster in a region with three or more AZs, you can select up to three AZs for multi-AZ instance deployment. The partition replicas of this instance will be forcibly distributed to the nodes in each AZ. This deployment mode allows your instance to normally provide services even if it is unavailable in a single AZ.
How multi-AZ deployment works
Pulsar implements cross-AZ disaster recovery based on rack awareness. Although the service nodes of different components are deployed in different racks in different AZs, they are essentially in a single Pulsar cluster.
As a type of ensemble placement policy, rack awareness is the algorithm used by BookKeeper client to select ensembles based on the network topology attribute.
How to select bookie
When a bookie cluster is deployed for the first time, each bookie will register a temporary zk-node to the current ZooKeeper, and the BookKeeper client will discover the bookie list via ZooKeeper. When a bookie changes or fails, the ensemble placement policy will be notified of this through the onClusterChanged (Set, Set) API and reconstruct a new network topology. Subsequent operations, such as newEnsemble (used to generate a group of bookies), will be performed based on the new network topology.
Network topology
Network topology shows the bookie node information in a cluster in a tree-like hierarchical structure. A bookie cluster may consist of many data centers in different regions. A data center contains bookie nodes distributed on different racks. In the tree-like structure, the leaf nodes show the bookie information.
Example 1: There are three bookies: bk1, bk2, and bk3 in region A. Their network locations are /region-a/rack-1/bk1, /region-a/rack-1/bk2, and /region-a/rack-2/bk3, respectively, as shown in the network topology below:
root
|
region-a
/ \\
rack-1 rack-2
/ \\\\
bk1 bk2 bk3
Example 2: There are four bookies: bk1, bk2, bk3, and bk4 in region A and region B. Their network locations are /region-a/rack-1/bk1, /region-a/rack-1/bk2, /region-b/rack-2/bk3, and /region-b/rack-2/bk4, respectively, as shown in the network topology below:
root
/ \\
region-a region-b
||
rack-1 rack-2
/ \\ / \\
bk1 bk2 bk3 bk4
The network locations are resolved with the DNSToSwitchMapping method from domain names or IP addresses. They must be in the format of "/region/rack", where the first slash (/) represents root, "region" represents data center, and "rack" represents rack information.
Note:
The DNSToSwitchMapping method implemented in Pulsar is called BookieRackAffinityMapping. It uses ZooKeeper to store rack information and only supports the rack awareness capability currently.
Use Limit
Currently, only pro clusters support multi-AZ deployment. For more information, see Product Selection.
Note
This feature is currently in beta testing and is free of charge.
Yes
No
Was this page helpful?