- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- Design of Data Table
- Importing Data
- Import Overview
- Import Methods
- Synchronizing Data with JDBC
- Synchronizing Data Through External Tables
- Real-time or Batch Writing of MySQL Data
- Importing Data from Kafka
- Importing with DataX
- Importing Doris from Logstash
- Data Transformation, Column Mapping and Filtering
- Importing Strict Mode
- Importing Data in JSON Format
- Exporting Data
- Basic Feature
- Query Optimization
- Ecological Expansion Feature
- Product Policy
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- Design of Data Table
- Importing Data
- Import Overview
- Import Methods
- Synchronizing Data with JDBC
- Synchronizing Data Through External Tables
- Real-time or Batch Writing of MySQL Data
- Importing Data from Kafka
- Importing with DataX
- Importing Doris from Logstash
- Data Transformation, Column Mapping and Filtering
- Importing Strict Mode
- Importing Data in JSON Format
- Exporting Data
- Basic Feature
- Query Optimization
- Ecological Expansion Feature
- Product Policy
Mutil Catalog Iceberg offers Doris the capability of accessing external Iceberg tables directly without tedious data import tasks, and give full play to its inherent OLAP capability to perform data analysis on Iceberg tables:
Iceberg data source can be connected to Doris.
It enables joint queries across Doris and the Iceberg data source, and thus allows more complex analysis.
This document mainly introduces how to use lceberg Catalog and preacautions during usage.
Note:
This feature is applicable to TCHouse-D 1.2 and later versions.
Iceberg V1/V2 table formats are supported, with V2 only supporting the Position Delete method, not supporting Equality Delete.
Creation Method
Data on HDFS:
CREATE RESOURCE ice_hms_resource PROPERTIES ('hive.metastore.uris' = 'thrift://172.16.xxxx:7004','type' = 'hms','dfs.nameservices'='HDFS1005116', 'dfs.ha.namenodes.HDFS1005116'='nn1,nn2','dfs.namenode.rpc-address.HDFS1005116.nn1'='172.16.xxxx:4007','dfs.namenode.rpc-address.HDFS1005116.nn2'='172.16.xxxx:4007','dfs.client.failover.proxy.provider.HDFS1005116'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider');
Data on COS:
CREATE CATALOG hive_cossab PROPERTIES ( "AWS_ENDPOINT" = "cos.ap-guangzhou.myqcloud.com", "AWS_REGION" = "ap-guangzhou", "AWS_SECRET_KEY" = "Wu9ByN6g4D8seHj0770jxxxxxxx", "use_path_style" = "true", "hive.metastore.uris" = "thrift://172.16.xxxx:7004", "type" = "hms", "AWS_ACCESS_KEY" = "AKIDaWJcCi9Rc4TqjV9hYHn9Nxxxxxxxxx" );
Type Matching
Supported column type of Iceberg corresponds to Doris as follows:
HMS Type | Doris Type | Comment |
boolean | boolean | - |
tinyint | tinyint | - |
smallint | smallint | - |
int | int | - |
bigint | bigint | - |
date | date | - |
timestamp | datetime | - |
float | float | - |
double | double | - |
char | char | - |
varchar | varchar | - |
decimal | decimal | - |
array<type> | array<type> | Nested arrays, such as array<array<int>>, are supported. |
map<KeyType, ValueType> | map<KeyType, ValueType> | Nested structure is not supported; KeyType and ValueType must be basic types. |
struct<col1: Type1, col2: Type2, ...> | struct<col1: Type1, col2: Type2, ...> | Nested structure is not supported; Type1, Type2, ... must be basic types. |
other | unsupported | - |
Time Travel
Reading a specified Snapshot from the Iceberg table is supported.
Every write operation on the Iceberg table will produce a new snapshot.
By default, read requests only read the snapshot of the latest version.
You can use the FOR TIME AS OF and FOR VERSION AS OF statements to read historical version data based on the snapshot ID or the time when the snapshot was generated as shown in the following examples:
SELECT * FROM iceberg_tbl FOR TIME AS OF "2022-10-07 17:20:37";SELECT * FROM iceberg_tbl FOR VERSION AS OF 868895038966572;
Query Usage
There is no difference from ordinary Doris OLAP tables.
select * from ice_catalog_name.database_name.table_name;
Yes
No
Was this page helpful?