tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Data Lake Compute
    Documentation
    Download PDF

    Procedure

    Last updated: 2024-08-07 17:29:12
    Download PDF

      Description

      Supported engine: SparkSQL
      Applicable table: External and native Iceberg tables

      Basic Statement

      CALL catalog_name.system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1);
      
      CALL catalog_name.system.procedure_name(arg_1, arg_2, ... arg_n);

      Snapshot Management

      rollback_to_snapshot

      Roll back the snapshot to a specified version. Input parameters: table name and version number.
      CALL `DataLakeCatalog`.`system`.rollback_to_snapshot('validation.dempts', 1);

      rollback_to_timestamp

      Roll back the snapshot to a specified timestamp. Input parameters: table name and timestamp
      CALL `DataLakeCatalog`.`system`.rollback_to_timestamp('validation.dempts', TIMESTAMP '2022-08-11 19:49:43.224');

      set_current_snapshot

      Set the current snapshot version. Input parameters: table name and version number.
      CALL `DataLakeCatalog`.`system`.set_current_snapshot('validation.dempts', 1);

      cherrypick_snapshot

      Cherrypick from a specified snapshot version to the current snapshot.
      CALL `DataLakeCatalog`.`system`.cherrypick_snapshot('validation.dempts', 1);
      CALL `DataLakeCatalog`.`system`.cherrypick_snapshot(snapshot_id => 1, table => 'my_table' )

      Metadata Management

      expire_snapshots

      Clean up expired snapshots to reduce the number of small files.
      CALL `Catalog`.`system`.expire_snapshots(table_name, [older_than], [retain_last], [max_concurrent_deletes], [stream_results]);
      Example:
      CALL `DataLakeCatalog`.`system`.expire_snapshots('validation.dempts', TIMESTAMP '2021-06-30 00:00:00.000', 100);

      remove_orphan_files

      Remove metadata files that are no longer referenced.
      CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
      Example:
      CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);
      CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');
      CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');

      remove_orphan_files

      Remove metadata files that are no longer referenced.
      CALL `Catalog`.`system`.remove_orphan_files(table_name, [older_than], [location], [dry_run], [max_concurrent_deletes]);
      Example:
      CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', dry_run=>TRUE);
      CALL `DataLakeCatalog`.`system`.remove_orphan_files(`table`=>'validation.dempts', `location`=>'cosn://channingdata-1305424723/example2/');
      CALL `DataLakeCatalog`.`system`.remove_orphan_files('validation.dempts', TIMESTAMP '2022-07-10 17:25:19.000');

      rewrite_data_files

      Merge and rewrite data files, i.e. merging small data files.
      CALL `Catalog`.`system`.rewrite_data_files(table_name, [strategy], [sort_order], [options], [where]);
      Example:
      CALL `DataLakeCatalog`.`system`.rewrite_data_files('validation.dempts');
      CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `strategy`=>'sort', `sort_order`=>'id DESC NULLS LAST,data ASC NULLS FIRST');
      CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `options`=>map('min-input-files','2'));
      CALL `DataLakeCatalog`.`system`.rewrite_data_files(`table`=>'validation.dempts', `where`=>'id = 3 and data = "foo"');

      rewrite_manifests

      Merge and rewrite manifests files.
      CALL `Catalog`.`system`.rewrite_manifests(table_name, [using_caching]);
      Example:
      CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts');
      CALL `DataLakeCatalog`.`system`.rewrite_manifests('validation.dempts', FALSE);

      ancestors_of

      Get the lineage information of a snapshot.
      CALL `Catalog`.`system`.ancestors_of(table_name, [snapshot_id]);
      Example:
      CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts');
      CALL `DataLakeCatalog`.`system`.ancestors_of('validation.dempts', 1);

      Data Table Migration Management

      Note
      The original table must be a Hive table or Spark table.

      snapshot

      Create lightweight temporary tables based on the original tables. The temporary tables directly reuse the snapshots of the original tables.
      CALL `Catalog`.`system`.snapshot(source_table, table, [location], [properties]);
      Example:
      CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap');
      CALL `DataLakeCatalog`.`system`.snapshot('validation.table_01', 'validation.snap2', 'cosn://channingdata-1305424723/example3/');

      migrate

      Update and replace table attributes.
      CALL `Catalog`.`system`.migrate(table, [properties]);
      Example:
      CALL `DataLakeCatalog`.`system`.migrate('validation.table_01');
      CALL `DataLakeCatalog`.`system`.migrate('validation.table_01', map('data', 'name'));

      add_files

      Load data files directly from Hive, and you can specify data files to a specific partition.
      CALL `Catalog`.`system`.add_files(table, source_table, [partition_filter]);
      Example:
      CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01');
      CALL `DataLakeCatalog`.`system`.add_files(`table`=>'validation.table_02', `source_table`=>'validation.table_01', `partition_filter`=>map('part_col', 'A'));
      
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy