Data Lake Compute
- Getting Started
- Operation Guide
- Console Operation Introduction
- Data Development and Exploration
- Data Exploration
- Data Query Task
- Resource Management
- Engine Management
- SuperSQL Engine
- Standard Engine
- Resource Group
- Private Connection
- Storage Configuration
- Ops Management
- Insight Management
- System Management
- User and Permission Management
- Monitoring and Alarms
- Development Guide
- System Restraints
- Client Access
- JDBC Access
- Practical Tutorial
- SQL Statement
- SuperSQL Statement
- Unified Statement
- DDL Statement
- ALTER TABLE
- DML Statement
- DQL Statement
- Iceberg Table Statement
- API Documentation
- Making API Requests
- Data Table APIs
- Task APIs
- Metadata APIs
- Service Configuration APIs
- Permission Management APIs
- Data Source Connection APIs
- Data Optimization APIs
- Data Engine APIs
- General Reference
- Operation Guide on Connecting Third-Party Software to DLC
Procedure
Last updated: 2024-08-07 17:29:12
Description
Supported engine: SparkSQL
Applicable table: External and native Iceberg tables
Basic Statement
CALL catalog_name.system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1);CALL catalog_name.system.procedure_name(arg_1, arg_2, ... arg_n);
Snapshot Management
rollback_to_snapshot
Roll back the snapshot to a specified version.
Input parameters: table name and version number.
CALL `DataLakeCatalog`.`system`.rollback_to_snapshot('validation.dempts', 1);
rollback_to_timestamp
Roll back the snapshot to a specified timestamp.
Input parameters: table name and timestamp
CALL `DataLakeCatalog`.`system`.rollback_to_timestamp('validation.dempts', TIMESTAMP '2022-08-11 19:49:43.224');
set_current_snapshot
Set the current snapshot version.
Input parameters: table name and version number.
CALL `DataLakeCatalog`.`system`.set_current_snapshot('validation.dempts', 1);
cherrypick_snapshot
Cherrypick from a specified snapshot version to the current snapshot.
CALL `DataLakeCatalog`.`system`.cherrypick_snapshot('validation.dempts', 1);CALL `DataLakeCatalog`.`system`.cherrypick_snapshot(snapshot_id => 1, table => 'my_table' )
Metadata Management
expire_snapshots
Clean up expired snapshots to reduce the number of small files.
CALL `Catalog`.`system`.expire_snapshots(table_name, [older_than], [retain_last], [max_concurrent_deletes], [stream_results]);
Example:
CALL `DataLakeCatalog`.`system`.expire_snapshots('validation.dempts', TIMESTAMP '2021-06-30 00:00:00.000', 100);
remove_orphan_files
Remove metadata files that are no longer referenced.
CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
Example:
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');
remove_orphan_files
Remove metadata files that are no longer referenced.
CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
Example:
CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');
rewrite_data_files
Merge and rewrite data files, i.e. merging small data files.
CALL `Catalog`.`system`.rewrite_data_files(table_name, [strategy], [sort_order], [options], [where]);
Example:
CALL `DataLakeCatalog`.`system`.rewrite_data_files('validation.dempts');CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `strategy`=>'sort', `sort_order`=>'id DESC NULLS LAST,data ASC NULLS FIRST');CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `options`=>map('min-input-files','2'));CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `where`=>'id = 3 and data = "foo"');
rewrite_manifests
Merge and rewrite manifests files.
CALL `Catalog`.`system`.rewrite_manifests(table_name, [using_caching]);
Example:
CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts');CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts', FALSE);
ancestors_of
Get the lineage information of a snapshot.
CALL `Catalog`.`system`.ancestors_of(table_name, [snapshot_id]);
Example:
CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts');CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts', 1);
Data Table Migration Management
Note
The original table must be a Hive table or Spark table.
snapshot
Create lightweight temporary tables based on the original tables. The temporary tables directly reuse the snapshots of the original tables.
CALL `Catalog`.`system`.snapshot(source_table, table, [location], [properties]);
Example:
CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap');CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap2', 'cosn://channingdata-1305424723/example3/');
migrate
Update and replace table attributes.
CALL `Catalog`.`system`.migrate(table, [properties]);
Example:
CALL `DataLakeCatalog`.`system`.migrate('validation.table_01');CALL `DataLakeCatalog`.`system`.migrate('validation.table_01', map('data', 'name'));
add_files
Load data files directly from Hive, and you can specify data files to a specific partition.
CALL `Catalog`.`system`.add_files(table, source_table, [partition_filter]);
Example:
CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01');CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01', `partition_filter`=>map('part_col', 'A'));
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No