- Product Introduction
- Purchase Guide
- Getting Started
- Complete Process for New User Activation
- DLC Data Import Guide
- Quick Start with Data Analytics in Data Lake Compute
- Quick Start with Permission Management in Data Lake Compute
- Quick Start with Partition Table
- Enabling Data Optimization
- Cross-Source Analysis of EMR Hive Data
- Standard Engine Configuration Guide
- Operation Guide
- Data Exploration
- Data Management
- Data Job
- Task History
- Insight Management
- Engine Management
- Data Engine Introduction
- SuperSQL Engine
- SuperSQL Engine Overview
- Purchasing Private Data Engine
- Renewing SuperSQL Engine
- Managing Private Data Engine
- Engine-Level Parameter Settings
- Disaster Recovery Cluster
- Engine Kernel Version
- Engine Network Configuration
- Associating Tag with Private Engine Resource
- Engine Local Cache
- Custom Task Scheduling Pool
- Standard Engine
- Ops Management
- Query Script Management
- Development Guide
- Client Access
- Practical Tutorial
- SQL Statement
- SuperSQL Statement
- Overview of SuperSQL Statement
- Unified Statement
- Common Data Types
- DDL Statement
- CREATE DATABASE
- SHOW DATABASES
- DESCRIBE DATABASE
- ALTER DATABASE
- DROP DATABASE
- CREATE TABLE
- REPLACE TABLE AS SELECT
- SHOW TABLES
- SHOW CREATE TABLE
- SHOW TBLPROPERTIES
- DESCRIBE TABLE
- SHOW COLUMNS IN TABLE
- ALTER TABLE
- ALTER TABLE ADD COLUMNS
- ALTER TABLE ADD COLUMN AFTER/FIRST
- ALTER TABLE DROP COLUMN
- ALTER TABLE ADD PARTATION
- SHOW PARTITIONS
- ALTER TABLE DROP PARTITION
- ALTER TABLE ADD PARTITION FIELD
- ALTER TABLE DROP PARTITION FIELD
- ALTER TABLE ... RENAME COLUMN
- ALTER TABLE SET TBLPROPERTIES
- ALTER TABLE SET LOCATION
- ALTER TABLE ... WRITE ORDERED BY
- ALTER TABLE ... WRITE DISTRIBUTED BY PARTITION
- ALTER TABLE ... SET IDENTIFIER FIELDS
- ALTER TABLE ... DROP IDENTIFIER FIELDS
- MSCK REPAIR TABLE
- ANALYZE TABLES
- DROP TABLE
- EXPLAIN
- CALL STATEMENT
- CREATE VIEW AS
- SHOW VIEWS
- DESCRIBE VIEW
- SHOW CREATE VIEW
- SHOW COLUMNS IN VIEW
- ALTER VIEW
- DROP VIEW
- CREATE FUNCTION
- SHOW FUNCTION
- DROP FUNCTION
- DML Statement
- DQL Statement
- Iceberg Table Statement
- Materialized View Statement
- SQL Implicit Conversion
- Functions
- Overview of Standard Spark Statement
- Overview of Standard Presto Statement
- Reserved Words
- SuperSQL Statement
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Data Table APIs
- Task APIs
- CreateSparkSessionBatchSQL
- CancelSparkSessionBatchSQL
- CancelTask
- CreateResultDownload
- CreateSparkAppTask
- CreateTask
- CreateTasks
- DeleteSparkApp
- DescribeEngineUsageInfo
- DescribeResultDownload
- DescribeSparkAppTasks
- DescribeTaskResult
- DescribeTasks
- QueryTaskCostDetail
- CreateSparkApp
- DescribeSparkAppJob
- DescribeSparkSessionBatchSqlLog
- ModifySparkApp
- DescribeSparkAppJobs
- ModifySparkAppBatch
- DescribeTaskStatistics
- DescribeQuery
- DescribeJobs
- DescribeJob
- Metadata APIs
- Service Configuration APIs
- CreateCHDFSBindingProduct
- DeleteCHDFSBindingProduct
- DescribeOtherCHDFSBindingList
- CreateStoreLocation
- DescribeStoreLocation
- ModifyDataEngineDescription
- RollbackDataEngineImage
- SwitchDataEngine
- SwitchDataEngineImage
- UpgradeDataEngineImage
- DeleteThirdPartyAccessUser
- DescribeDataEngineImageVersions
- DescribeSubUserAccessPolicy
- DescribeThirdPartyAccessUser
- RegisterThirdPartyAccessUser
- RestartDataEngine
- UpdateUserDataEngineConfig
- UpdateDataEngineConfig
- Permission Management APIs
- AddUsersToWorkGroup
- AttachUserPolicy
- AttachWorkGroupPolicy
- BindWorkGroupsToUser
- CreateUser
- CreateWorkGroup
- DeleteUser
- DeleteUsersFromWorkGroup
- DeleteWorkGroup
- DescribeUserInfo
- DescribeUserRoles
- DescribeUserType
- DescribeUsers
- DescribeWorkGroupInfo
- DescribeWorkGroups
- DetachUserPolicy
- DetachWorkGroupPolicy
- ModifyUser
- ModifyUserType
- ModifyWorkGroup
- UnbindWorkGroupsFromUser
- UpdateRowFilter
- CheckGrantedPermission
- Database APIs
- Data Source Connection APIs
- Data Optimization APIs
- Data Engine APIs
- Data Types
- Error Codes
- General Reference
- DLC Policy
- Service Level Agreement
- Contact Us
- Product Introduction
- Purchase Guide
- Getting Started
- Complete Process for New User Activation
- DLC Data Import Guide
- Quick Start with Data Analytics in Data Lake Compute
- Quick Start with Permission Management in Data Lake Compute
- Quick Start with Partition Table
- Enabling Data Optimization
- Cross-Source Analysis of EMR Hive Data
- Standard Engine Configuration Guide
- Operation Guide
- Data Exploration
- Data Management
- Data Job
- Task History
- Insight Management
- Engine Management
- Data Engine Introduction
- SuperSQL Engine
- SuperSQL Engine Overview
- Purchasing Private Data Engine
- Renewing SuperSQL Engine
- Managing Private Data Engine
- Engine-Level Parameter Settings
- Disaster Recovery Cluster
- Engine Kernel Version
- Engine Network Configuration
- Associating Tag with Private Engine Resource
- Engine Local Cache
- Custom Task Scheduling Pool
- Standard Engine
- Ops Management
- Query Script Management
- Development Guide
- Client Access
- Practical Tutorial
- SQL Statement
- SuperSQL Statement
- Overview of SuperSQL Statement
- Unified Statement
- Common Data Types
- DDL Statement
- CREATE DATABASE
- SHOW DATABASES
- DESCRIBE DATABASE
- ALTER DATABASE
- DROP DATABASE
- CREATE TABLE
- REPLACE TABLE AS SELECT
- SHOW TABLES
- SHOW CREATE TABLE
- SHOW TBLPROPERTIES
- DESCRIBE TABLE
- SHOW COLUMNS IN TABLE
- ALTER TABLE
- ALTER TABLE ADD COLUMNS
- ALTER TABLE ADD COLUMN AFTER/FIRST
- ALTER TABLE DROP COLUMN
- ALTER TABLE ADD PARTATION
- SHOW PARTITIONS
- ALTER TABLE DROP PARTITION
- ALTER TABLE ADD PARTITION FIELD
- ALTER TABLE DROP PARTITION FIELD
- ALTER TABLE ... RENAME COLUMN
- ALTER TABLE SET TBLPROPERTIES
- ALTER TABLE SET LOCATION
- ALTER TABLE ... WRITE ORDERED BY
- ALTER TABLE ... WRITE DISTRIBUTED BY PARTITION
- ALTER TABLE ... SET IDENTIFIER FIELDS
- ALTER TABLE ... DROP IDENTIFIER FIELDS
- MSCK REPAIR TABLE
- ANALYZE TABLES
- DROP TABLE
- EXPLAIN
- CALL STATEMENT
- CREATE VIEW AS
- SHOW VIEWS
- DESCRIBE VIEW
- SHOW CREATE VIEW
- SHOW COLUMNS IN VIEW
- ALTER VIEW
- DROP VIEW
- CREATE FUNCTION
- SHOW FUNCTION
- DROP FUNCTION
- DML Statement
- DQL Statement
- Iceberg Table Statement
- Materialized View Statement
- SQL Implicit Conversion
- Functions
- Overview of Standard Spark Statement
- Overview of Standard Presto Statement
- Reserved Words
- SuperSQL Statement
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Data Table APIs
- Task APIs
- CreateSparkSessionBatchSQL
- CancelSparkSessionBatchSQL
- CancelTask
- CreateResultDownload
- CreateSparkAppTask
- CreateTask
- CreateTasks
- DeleteSparkApp
- DescribeEngineUsageInfo
- DescribeResultDownload
- DescribeSparkAppTasks
- DescribeTaskResult
- DescribeTasks
- QueryTaskCostDetail
- CreateSparkApp
- DescribeSparkAppJob
- DescribeSparkSessionBatchSqlLog
- ModifySparkApp
- DescribeSparkAppJobs
- ModifySparkAppBatch
- DescribeTaskStatistics
- DescribeQuery
- DescribeJobs
- DescribeJob
- Metadata APIs
- Service Configuration APIs
- CreateCHDFSBindingProduct
- DeleteCHDFSBindingProduct
- DescribeOtherCHDFSBindingList
- CreateStoreLocation
- DescribeStoreLocation
- ModifyDataEngineDescription
- RollbackDataEngineImage
- SwitchDataEngine
- SwitchDataEngineImage
- UpgradeDataEngineImage
- DeleteThirdPartyAccessUser
- DescribeDataEngineImageVersions
- DescribeSubUserAccessPolicy
- DescribeThirdPartyAccessUser
- RegisterThirdPartyAccessUser
- RestartDataEngine
- UpdateUserDataEngineConfig
- UpdateDataEngineConfig
- Permission Management APIs
- AddUsersToWorkGroup
- AttachUserPolicy
- AttachWorkGroupPolicy
- BindWorkGroupsToUser
- CreateUser
- CreateWorkGroup
- DeleteUser
- DeleteUsersFromWorkGroup
- DeleteWorkGroup
- DescribeUserInfo
- DescribeUserRoles
- DescribeUserType
- DescribeUsers
- DescribeWorkGroupInfo
- DescribeWorkGroups
- DetachUserPolicy
- DetachWorkGroupPolicy
- ModifyUser
- ModifyUserType
- ModifyWorkGroup
- UnbindWorkGroupsFromUser
- UpdateRowFilter
- CheckGrantedPermission
- Database APIs
- Data Source Connection APIs
- Data Optimization APIs
- Data Engine APIs
- Data Types
- Error Codes
- General Reference
- DLC Policy
- Service Level Agreement
- Contact Us
DLC offers two types of engines: the Standard Engine and the SuperSQL Engine. For a detailed comparison, see the table below or see the Data Engine Introduction. You can select the appropriate engine based on your specific business needs. If you choose the Standard Engine, you can follow the instructions in this document for configuration and usage.
Engine Types | Available Types | Main Features | Usage Requirements | Purchase Recommendations |
Standard Engine | Spark
Presto | Integrated Spark: The Standard Spark Engine supports native syntax from the Spark/Presto community, making it easy to learn and migrate. Flexible usage: Supports both Hive JDBC and Presto JDBC. Integrated Spark: The Standard Spark Engine can execute SQL and Spark batch tasks. | 1. Requires the use of Spark/Presto native syntax. 2. Prefer to purchase a Spark engine for batch jobs and offline SQL tasks. 3. Prefer to use Hive JDBC and Presto JDBC. | |
SuperSQL Engine | SparkSQL\\nSpark Jobs\\nPresto | Unified syntax: A single syntax is applicable to both Spark and Presto engines. Supports federated queries. | Requires learning the SuperSQL unified syntax.\\nFor SQL/batch tasks, it is recommended to purchase the corresponding engine type. | 1. Prefer to use Spark + Presto unified syntax. 2. Federated queries are required. |
Note:
1. Before purchasing, you should ensure that your account has been granted financial permissions in CAM.
2. Resources cannot be used across regions, so confirm that the current region is correct before purchasing.
Standard Engine Configuration Guide
After completing the purchase and configuration of the Standard Engine, you can use it within DLC's Data Exploration. Additionally, for the Spark Standard Engine, if you have multi-tenant or task isolation requirements, you can also configure Resource Group for resource allocation and isolation. The detailed guide is as follows:
Step 1: Purchasing the Engine
Note:
1. Engines cannot be used across regions.
2. Engine specification recommendation: Since a 16 CU cluster is relatively small, it is recommended only for testing scenes. For real production environments, it is recommended to choose a cluster with a specification of 64 CUs or more.
3. Engine network configuration: Custom network configurations can be set during the initial purchase. If you need to make changes later, please Submit Ticket to apply for modifications.
Step 2: Using Data Exploration
Selecting the Standard Engine for Queries
Note:
Depending on the type of Standard Engine, you may need to switch to the corresponding syntax for queries.
If you select the Standard Spark Engine in Data Exploration, you can allocate task resources by using the DLC default resource group, a created resource group, or a one-time resource group (custom configuration).
Retrieving Full Results
Currently, the Standard Engine only supports returning up to 1,000 query results in the console. To retrieve the full results, you can see the following methods:
Engine | Retrieval Method |
Standard Spark Engine | 1. Users can configure the engine to automatically save query results to a COS path or view them in DLC's managed storage. 2. Results can be downloaded locally for review. |
Standard Presto Engine | Retrieve full results via JDBC. |
Step 3: Configuring Resource Groups (Optional)
Resource groups provide a secondary queue division of computing resources within the Spark Standard Engine. For a detailed introduction, see Resource Group Introduction. The computing units (CUs) of the DLC Spark Standard Engine can be allocated across multiple resource groups as needed. You can set the minimum and maximum CU limits for each resource group, along with start/stop policies, concurrency levels, and dynamic/static parameters, ensuring resource isolation and efficient workload management in complex scenes such as multi-tenancy and multi-tasking.
When you purchase a Standard Spark Engine, DLC provides a default resource group and also allows you to create multiple custom resource groups based on your specific business needs for flexible usage.
Note:
An engine can have a one-to-many relationship with resource groups. For example, Engine A can have several resource groups.
Managing and Configuring Resource Groups
1. Click to enter the resource group management of the corresponding engine.
2. Enter the Resource Management Group interface, and click Create Resource Group to configure a custom resource group. Alternatively, you can view and use the DLC default-configured resource group (no configuration required).
Appendix
Recommendations for Selecting Gateway Specifications
Gateway Specification | Spark Batch Instant Concurrency (Submitted/Running Tasks) | Concurrent Spark SQL/Presto SQL Queries | Number of Presto Engines Managed | Number of Spark Resource Groups Managed | Gateway HA |
2 CU | 30/50 | 100 | 4 | 50 | No |
16 CU | 80/150 | 250 | 12 | 150 | Yes |
32 CU | 220/400 | 600 | 35 | 400 | Yes |
64 CU | 400/600 | 1000 | 70 | 700 | Yes |
Note:
The gateway is provided by default with a 2 CU specification (free of charge). If you need to upgrade the specifications, you can click Gateway details → select Specification Configuration to adjust and purchase.
Yes
No
Was this page helpful?