The resource group is a secondary queue division of the computing resources within a Standard Spark Engine. Resource groups belong to a parent Standard Engine, and resource groups under the same engine share resources with each other. The computing units (CUs) of the DLC Standard Spark Engine can be allocated to multiple resource groups as needed. You can configure each resource group's minimum and maximum CU limits, start and stop policies, concurrency, and dynamic/static parameters to efficiently manage resource isolation and workload in complex scenes such as multi-tenancy and multi-tasking.
For example, you can create separate resource groups within a Standard Spark Engine, such as a Report Resource Group, a Data Warehouse Resource Group, and a Historical Backfill Resource Group. You can set the upper and lower limits of computing units (CUs) for each resource group and assign relevant SQL tasks or jobs, such as reports and data warehouse tasks, to the appropriate resource group, ensuring resource isolation between different types of tasks and preventing individual large queries from monopolizing resources for extended periods.
Features
Resource Group Isolation
Resource groups enable resource isolation within the Standard Spark Engine. You can assign specific resource groups to different users or queries, effectively isolating resources and preventing a single user or large query from monopolizing most of the computing engine's resources.
Resource Group Elasticity
By configuring the number of Executors in a resource group for dynamic allocation, the resource group can adjust the resources used by SQL tasks or jobs based on the workload, effectively improving resource utilization.
The dynamic allocation configuration is shown in the diagram below:
Both Task 01 and Task 02 are set to dynamic allocation, each using 8 CUs at Time A. By Time B, Task 01 only requires 4 CUs, releasing 4 CUs of idle resources for Task 02 to use, thereby improving overall resource utilization. This process is illustrated in the diagram below:
Usage Limitations
The resource group name should be globally unique. It is recommended to use an all-English name.
Terminology
Description
Illustration
Default Resource Groups
(System created by default)
Exist upon engine creation, and named as default-rg-xxx.
The default resource group starts in a suspended status, with settings for automatic start and automatic suspension.
The default resource group supports modification of resource configurations.
The default resource group supports configuring start/stop policies, setting concurrency limits, and adjusting dynamic/static parameters.
The default resource group cannot be deleted.
(User manually created)
The custom resource group supports the modification of resource configurations.
The custom resource group supports configuring start/stop policies, setting concurrency limits, and adjusting dynamic/static parameters.
