Data Lake Compute
- Getting Started
- Operation Guide
- Console Operation Introduction
- Data Development and Exploration
- Data Exploration
- Data Query Task
- Resource Management
- Engine Management
- SuperSQL Engine
- Standard Engine
- Resource Group
- Private Connection
- Storage Configuration
- Ops Management
- Insight Management
- System Management
- User and Permission Management
- Monitoring and Alarms
- Development Guide
- System Restraints
- Client Access
- JDBC Access
- Practical Tutorial
- SQL Statement
- SuperSQL Statement
- Unified Statement
- DDL Statement
- ALTER TABLE
- DML Statement
- DQL Statement
- Iceberg Table Statement
- API Documentation
- Making API Requests
- Data Table APIs
- Task APIs
- Metadata APIs
- Service Configuration APIs
- Permission Management APIs
- Data Source Connection APIs
- Data Optimization APIs
- Data Engine APIs
- General Reference
- Operation Guide on Connecting Third-Party Software to DLC
Standard Engine Configuration Guide
Last updated: 2024-09-04 11:11:02
DLC offers two types of engines: the Standard Engine and the SuperSQL Engine. For a detailed comparison, see the table below or see the Data Engine Introduction. You can select the appropriate engine based on your specific business needs. If you choose the Standard Engine, you can follow the instructions in this document for configuration and usage.
Engine Types | Available Types | Main Features | Usage Requirements | Purchase Recommendations |
Standard Engine | Spark
Presto | Integrated Spark: The Standard Spark Engine supports native syntax from the Spark/Presto community, making it easy to learn and migrate. Flexible usage: Supports both Hive JDBC and Presto JDBC. Integrated Spark: The Standard Spark Engine can execute SQL and Spark batch tasks. | 1. Requires the use of Spark/Presto native syntax. 2. Prefer to purchase a Spark engine for batch jobs and offline SQL tasks. 3. Prefer to use Hive JDBC and Presto JDBC. | |
SuperSQL Engine | SparkSQL\\nSpark Jobs\\nPresto | Unified syntax: A single syntax is applicable to both Spark and Presto engines. Supports federated queries. | Requires learning the SuperSQL unified syntax.\\nFor SQL/batch tasks, it is recommended to purchase the corresponding engine type. | 1. Prefer to use Spark + Presto unified syntax. 2. Federated queries are required. |
Note:
1. Before purchasing, you should ensure that your account has been granted financial permissions in CAM.
2. Resources cannot be used across regions, so confirm that the current region is correct before purchasing.
Standard Engine Configuration Guide
After completing the purchase and configuration of the Standard Engine, you can use it within DLC's Data Exploration. Additionally, for the Spark Standard Engine, if you have multi-tenant or task isolation requirements, you can also configure Resource Group for resource allocation and isolation. The detailed guide is as follows:
Step 1: Purchasing the Engine
Note:
1. Engines cannot be used across regions.
2. Engine specification recommendation: Since a 16 CU cluster is relatively small, it is recommended only for testing scenes. For real production environments, it is recommended to choose a cluster with a specification of 64 CUs or more.
3. Engine network configuration: Custom network configurations can be set during the initial purchase. If you need to make changes later, please Submit Ticket to apply for modifications.
Step 2: Using Data Exploration
Selecting the Standard Engine for Queries
Note:
Depending on the type of Standard Engine, you may need to switch to the corresponding syntax for queries.
If you select the Standard Spark Engine in Data Exploration, you can allocate task resources by using the DLC default resource group, a created resource group, or a one-time resource group (custom configuration).
Retrieving Full Results
Currently, the Standard Engine only supports returning up to 1,000 query results in the console. To retrieve the full results, you can see the following methods:
Engine | Retrieval Method |
Standard Spark Engine | 1. Users can configure the engine to automatically save query results to a COS path or view them in DLC's managed storage. 2. Results can be downloaded locally for review. |
Standard Presto Engine | Retrieve full results via JDBC. |
Step 3: Configuring Resource Groups (Optional)
Resource groups provide a secondary queue division of computing resources within the Spark Standard Engine. For a detailed introduction, see Resource Group Introduction. The computing units (CUs) of the DLC Spark Standard Engine can be allocated across multiple resource groups as needed. You can set the minimum and maximum CU limits for each resource group, along with start/stop policies, concurrency levels, and dynamic/static parameters, ensuring resource isolation and efficient workload management in complex scenes such as multi-tenancy and multi-tasking.
When you purchase a Standard Spark Engine, DLC provides a default resource group and also allows you to create multiple custom resource groups based on your specific business needs for flexible usage.
Note:
An engine can have a one-to-many relationship with resource groups. For example, Engine A can have several resource groups.
Managing and Configuring Resource Groups
1. Click to enter the resource group management of the corresponding engine.
2. Enter the Resource Management Group interface, and click Create Resource Group to configure a custom resource group. Alternatively, you can view and use the DLC default-configured resource group (no configuration required).
Appendix
Recommendations for Selecting Gateway Specifications
Gateway Specification | Spark Batch Instant Concurrency (Submitted/Running Tasks) | Concurrent Spark SQL/Presto SQL Queries | Number of Presto Engines Managed | Number of Spark Resource Groups Managed | Gateway HA |
2 CU | 30/50 | 100 | 4 | 50 | No |
16 CU | 80/150 | 250 | 12 | 150 | Yes |
32 CU | 220/400 | 600 | 35 | 400 | Yes |
64 CU | 400/600 | 1000 | 70 | 700 | Yes |
Note:
The gateway is provided by default with a 2 CU specification (free of charge). If you need to upgrade the specifications, you can click Gateway details → select Specification Configuration to adjust and purchase.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No