The status of the Consumer Group on the consumer group list page mainly includes several types: Dead, Empty, PreparingRebalance, AwaitingSync, and Stable. Among them, the most common are the three statuses of Empty, Stable, and Dead. The state machine switch in the Consumer Group is shown in the figure below:
Dead: Consumer group has no members and Metadata has been removed.
Empty: No members in the consumption group currently. If all offsets in the group have expired, it will become Dead. Newly created Group is in Empty status by default.
According to the regulations of open source Kafka 0.10.x, if there are no members in the consumption group and the status lasts for more than 7 days, the consumer group will be automatically deleted.
Stable: All consumers in the group have joined, and the group is in a stable status.
Rebalance Status Details
Cause of Rebalance Occurrence
According to the Consumer Group's state machine, the Group may perform Rebalance when the Consumer Group is in Empty, AwaitingSync, or Stable status. Rebalance may occur in the following scenarios:
A consumer subscribes to a topic.
A consumer is disabled.
When a Consumer is considered in the Dead state by the Group Coordinator (Coordinator).
If a Consumer does not send a heartbeat to the Group Coordinator within session.timeout.ms, the Consumer will be in the Dead state and a Rebalance will be initiated. For more information, please refer to CKafka Common Parameter Configuration Guide.
The number of partitions increases.
Subscribe to a non-existent Topic.
If you subscribe to a Topic that has not been created yet, Rebalance will occur when the Topic is created; likewise, if a Topic that has already been subscribed to is deleted, Rebalance will also occur.
Application crashes.
Rebalance Process Analysis
Taking the mechanism of Kafka 0.10 version as an example, the Rebalance process is analyzed as follows:
1. When any Consumer wants to join a Consumer Group, it will send a JoinGroup request to the Group Coordinator. The first Consumer to join the Group will become the Group Leader.
2. The Leader will receive the list of all consumers in the Group from the Group Coordinator and assign partitions to consumers in the Group. partition assignment can be implemented through the PartitionAssignor API.
3. After the assignment is done, the Leader will send the assignment result to the Group Coordinator, and the Coordinator will send the result to all Consumers.
4. Therefore, each Consumer can only view the partition assigned to itself, and the Leader is the only Consumer that can get the Consumers in the Consumer Group and their partition details.
The above process will be executed once whenever Rebalance occurs.
