- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- Design of Data Table
- Importing Data
- Import Overview
- Import Methods
- Synchronizing Data with JDBC
- Synchronizing Data Through External Tables
- Real-time or Batch Writing of MySQL Data
- Importing Data from Kafka
- Importing with DataX
- Importing Doris from Logstash
- Data Transformation, Column Mapping and Filtering
- Importing Strict Mode
- Importing Data in JSON Format
- Exporting Data
- Basic Feature
- Query Optimization
- Ecological Expansion Feature
- Product Policy
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Development Guide
- Design of Data Table
- Importing Data
- Import Overview
- Import Methods
- Synchronizing Data with JDBC
- Synchronizing Data Through External Tables
- Real-time or Batch Writing of MySQL Data
- Importing Data from Kafka
- Importing with DataX
- Importing Doris from Logstash
- Data Transformation, Column Mapping and Filtering
- Importing Strict Mode
- Importing Data in JSON Format
- Exporting Data
- Basic Feature
- Query Optimization
- Ecological Expansion Feature
- Product Policy
Note:
The content showcased in this document is only suitable for TCHouse-D v1.1 and below. For later versions, it is recommended to use the Multi-Catalog feature for interfacing with external data directories.
The Hudi External Table of Doris provides Doris the capability to directly access Hudi external tables. This eliminates the tedious work of data importing, and leverages OLAP capabilities of Doris for data analysis in Hudi tables:
1. Doris can access Hudi data source.
2. Supports joint queries of table in Doris and Hive, the Hudi data source for more complex analysis.
This document mainly introduces the usage and considerations of this feature.
Usage
Creating Hudi External Tables in Doris
You can create Hudi external tables in Doris in the following two ways. There is no need to claim the definition of column when creating the table, for Doris can obtain column information from HiveMetaStore during query.
1. Create an independent external table for mounting the Hudi table. For the related syntax, see CREATE TABLE.
-- SyntaxCREATE [EXTERNAL] TABLE table_name[(column_definition1[, column_definition2, ...])]ENGINE = HUDI[COMMENT "comment"]PROPERTIES ("hudi.database" = "hudi_db_in_hive_metastore","hudi.table" = "hudi_table_in_hive_metastore","hudi.hive.metastore.uris" = "thrift://127.0.0.1:9083");-- Example: Mount the hudi_table_in_hive_metastore under the hudi_db_in_hive_metastore in HiveMetaStore. The schema is not specified during mounting.CREATE TABLE `t_hudi`ENGINE = HUDIPROPERTIES ("hudi.database" = "hudi_db_in_hive_metastore","hudi.table" = "hudi_table_in_hive_metastore","hudi.hive.metastore.uris" = "thrift://127.0.0.1:9083");-- Example: Specify the schema during mountingCREATE TABLE `t_hudi` (`id` int NOT NULL COMMENT "id number",`name` varchar(10) NOT NULL COMMENT "user name") ENGINE = HUDIPROPERTIES ("hudi.database" = "hudi_db_in_hive_metastore","hudi.table" = "hudi_table_in_hive_metastore","hudi.hive.metastore.uris" = "thrift://127.0.0.1:9083");
Parameter description:
External Table Column:
The column name can be left unspecified. In this case, Doris will obtain the column information from HiveMetaStore during the query. This method of table creation is recommended.
The name of the specified column must exist in the Hudi table.
The ENGINE must be specified to HUDI.
PROPERTIES attributes:
hudi.hive.metastore.uris: The address for the Hive Metastore service.hudi.database: The name of the corresponding database when Hudi is mounted.hudi.table: The name of the corresponding table when Hudi is mounted.Display table structure
You can view the table structure with
SHOW CREATE TABLE peek .Type Matching
The following table shows the corresponding relationships between supported Hudi column types and Doris:
Hudi | Doris | Description |
BOOLEAN | BOOLEAN | - |
INTEGER | INT | - |
LONG | BIGINT | - |
FLOAT | FLOAT | - |
DOUBLE | DOUBLE | - |
DATE | DATE | - |
TIMESTAMP | DATETIME | Converting timestamp to datetime may cause loss of precision |
STRING | STRING | - |
UUID | VARCHAR | Use VARCHAR as a substitute |
DECIMAL | DECIMAL | - |
TIME | - | Not supported |
FIXED | - | Not supported |
BINARY | - | Not supported |
STRUCT | - | Not supported |
LIST | - | Not supported |
MAP | - | Not supported |
Note
Currently, the supported Hudi version is 0.10.0 by default. Other versions haven't been tested.
Query Usage
You can use the Hudi external table in Doris the same way as using Doris OLAP tables, except that the Doris data models (Rollup, Pre-Aggregation, and Materialized Views, etc.) are unavailable.
select * from t_hudi where k1 > 1000 and k3 ='term' or k4 like '%doris';
Yes
No
Was this page helpful?