Practical Tutorial on Setting Load-based Scaling Rules

Recent Pages

Practical Tutorial on Setting Load-based Scaling Rules

Last updated: 2024-10-30 12:33:14

Based on the changes in YARN metrics within the cluster, select the metrics from the past that align with business fluctuations, configure specific thresholds, and then save and apply the settings. Once business activity changes, the corresponding rules will be triggered. The selected metrics should be inversely related to capacity changes, so after scaling activities occur, the change in instance numbers can reduce the corresponding metrics.
Example:
To configure the scale-out rule: if the average value of AppsPending#root is greater than or equal to 1 within 300 seconds and this occurs consecutively 2 times, a scale-out action will be triggered. This can effectively reduce the number of pending tasks in the queue.
Scaling-up rule
Scaling-down rule similarly: configure based on actual conditions. For more details on other rule configuration items and usage instructions, see Set Load Scaling.
1.1 For each rule, you can configure multiple metric conditions, and when all the conditions are met, scaling is triggered.
1.2 To avoid frequent scaling activities that lead to resource waste, you can configure a cooling period for the rule. During the cooling period, even if the scaling conditions are met, no scaling activity will occur.
1.3 Configure an active time (the current rule takes effect within a customized time range). Different scaling rules are allowed to be combined, and you can set different scaling conditions for different time periods.
Note:
1. During peak periods, resource contention may prevent the actual scale-out number from reaching the elastic target number of machines. It is recommended to enable the Resource supplement retry for your scale-out rule.
2. When the scale-in action is triggered, nodes may still be executing tasks. To avoid immediate release of the nodes, it is recommended that you enable graceful scale-in. For more details, see Graceful Scale-In.
﻿