Tencent Cloud WeData
- Product Introduction
- Purchase Guide
- Execute Resource Purchase Description
- Preparations
- Add allowlist /security groups Overview (Optional)
- Operation Guide
- Console Operation
- Execution Resource Group Configuration
- Approval Management
- General Configuration
- Project Management
- Project Execution Resource Group
- Data Integration
- Integration resource configuration and management
- Real-time Synchronization Task Configuration and Operation and Maintenance
- Data Sources Supported by Real-time Synchronization
- Data Source List
- Whole Database Synchronization Task Configuration
- Single Table Synchronization Task Configuration
- Real-time synchronization operation and maintenance
- Real-time OPS
- Monitoring and Alarms
- Offline Synchronization Task Configuration and Operation and Maintenance
- Data Sources Supported by Offline Synchronization
- Data Source List
- DataInLong Offline Synchronization Configuration and Operation and Maintenance
- Data Development：Offline Synchronization, Configuration and Ops
- Offline Synchronization and Ops
- Practical Tutorial
- Data Development
- Engine User Guide
- Task Development
- Task Node Type
- General Node
- Task Scheduling Configuration
- Development Space
- Task Operation and Maintenance
- Data Analysis
- Data Assets
- Data Map
- Metadata Collection
- Metadata Management
- My Data
- My Responsibilities
- Managed by Me
- Configuration Management
- Data Security
- Data protection
- API Documentation
- Making API Requests
- Smart Ops Related Interfaces
- Data Development APIs
- Data Operations Related Interfaces
- Metadata Related Interfaces
- Task Operation and Maintenance APIs
- Instance Operation and Maintenance Related Interfaces
- Data Map and Data Dictionary APIs
- Data Quality Related Interfaces
- DataInLong APIs
- Platform management related APIs
- Data Source Management APIs
- Asset Data APIs
- Related Agreement
DocumentationTencent Cloud WeDataOperation GuideData IntegrationAutomatic Table Creation Capability of Synchronization Tasks
Automatic Table Creation Capability of Synchronization Tasks
Last updated: 2024-11-01 17:33:23
Automatic Table Creation Capability of Synchronization Tasks
Last updated: 2024-11-01 17:33:23
DataInLong provides one-click table creation, automatic table creation, and other features in offline and real-time scenarios to help you quickly build target data tables before or during synchronization, thereby improving data link usage efficiency. This article mainly introduces the usage of the corresponding features, supported links, type conversion relationships between core heterogeneous data sources, etc.
Feature Introduction and Use Cases
DataInLong offers three table creation methods: one-click table creation, batch table creation, and automatic table creation based on DDL messages:
Feature Items | Feature Illustration | Feature Introduction | Use cases |
One-Click Table Creation |  | This feature can automatically convert multiple table objects specified from the source end to the target end DDL, supporting users to manually modify the converted DDL content and create them in batches at once.
Feature Keywords: Heterogeneous DDL Automatic Conversion, Batch Table Creation, Manual Table Creation. | This feature is suitable for quickly constructing target tables resembling the source table structure for the source end's inventory. For complex table models that require customized business definitions, users can edit the automatically generated DDL statements based on business features to enhance heterogeneous DDL coding efficiency. |
Batch Table Creation |  | | |
Automatic Table Creation Based on Source Side DDL Messages |  | This feature can monitor in real time whether new table DDL exists at the source end. Once an identifiable new table message is generated at the source end, the target end can automatically generate target table objects based on the source table structure. feature keywords: DDL automatic detection and response, automatic table creation. | This feature is suitable for data synchronization scenarios where it is desired to keep the structure of the target end consistent with the source end in real-time during real-time database migration. |
Supported Links for One-Click Table Creation
Offline single table tasks and real-time single table tasks support the one-click creation of target tables based on source tables, mainly for scenarios of adding a small number of new database tables to the lake or warehouse.
Offline Single Table Synchronization Task
Target Number | Target | Supported Source Ends |
1 | Doris | MySQL,TDSQL-C MySQL,PostgreSQL |
2 | Hive | MySQL,Oracle,SQL Server,OceanBase,PostgreSQL |
3 | Iceberg | MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng |
4 | TCHouse-P | MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng |
5 | DLC | MySQL,TDSQL-C MySQL,PostgreSQL,Elasticsearch,Kafka |
Real-time Single Table Synchronization Task
Target Number | Target | Supported Source Ends |
1 | TCHouse-P | MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng |
2 | Iceberg | MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, PostgreSQL, SQL Server, Oceanbase, Dameng |
3 | Doris | MySQL,TDSQL-C MySQL,PostgreQL |
4 | DLC | MySQL,TDSQL-C MySQL,PostgreSQL,Elasticsearch,Kafka |
Supported Links for Batch Table Creation
Offline batch creation of single table tasks and real-time whole database tasks support bulk creation of target tables based on source tables during the task configuration stage, mainly for scenarios of ingesting a large number of existing database tables into the lake or warehouse.
Offline Batch Sync Task
Target Number | Target | Supported Source Ends |
1 | Hive | MySQL,Oracle,SQL Server,PostgreSQL |
2 | Iceberg | MySQL, TDSQL-C for MySQL, TDSQL for MySQL, Oracle, SQL Server, PostgreSQL, Dameng |
3 | TCHouse-P | MySQL, TDSQL-C for MySQL, TDSQL for MySQL, Oracle, PostgreSQL, OceanBase, Dameng |
4 | Doris | MySQL,TDSQL-C MySQL,PostgreSQL |
5 | DLC | MySQL,TDSQL-C MySQL,PostgreSQL |
Real-time Full Database Migration Task
Target Number | Target | Supported Source Ends |
1 | Doris | MySQL,TDSQL-C MySQL,TDSQL MySQL |
2 | DLC | MySQL,TDSQL-C MySQL,TDSQL MySQL |
Support Automatic Table Creation Based on Source Side DDL Messages
Except for the following links, all other real-time full database links support automatic creation of target databases and tables during task running based on source side DDL messages:
Serial number | Source | Target | Remarks |
1 | MySQL | PostgreSQL | Only supports using existing databases, schemas, and tables |
2 | OceanBase | Doris,Iceberg | Only supports using existing databases and tables |
3 | Kafka | Doris | Only supports using existing databases and tables |
Core Heterogeneous Data Source Table Creation Type Conversion Relationship
MySQL to Doris Table Field Type Conversion:
| MySQL (source end) | DORIS (target end) | Additional Notes | ||
| Data Type | Precision Description | Data Type | Precision Description | |
| BOOLEAN | 0 represents false, 1 represents true | BOOLEAN | 0 represents false, 1 represents true | - |
Value Type | TINYINT | Range [-128, 127] | TINYINT | Range [-128, 127] | - |
SMALLINT | Range [-32768, 32767] | SMALLINT | Range [-32768, 32767] | - | |
MEDIUMINT | Range [-8388608, 8388607] | INT | Range [-2147483648, 2147483647] | - | |
INT | Range [-2147483648, 2147483647] | INT | Range [-2147483648, 2147483647] | - | |
BIGINT | Range [-9223372036854775808, 9223372036854775807] | BIGINT | Range [-9223372036854775808, 9223372036854775807] | - | |
UNSIGNED TINYINT | Range [0, 255] | SMALLINT | Range [-32768, 32767] | - | |
UNSIGNED MEDIUMINT | Range [0, 16777215] | INT | Range [-2147483648, 2147483647] | - | |
UNSIGNED INT | Range [0, 4294967295] | BIGINT | Range [-9223372036854775808, 9223372036854775807] | - | |
UNSIGNED BIGINT | Range [0, 18,446,744,073,709,551,615] | LARGEINT | Range [-2^127 + 1 ~ 2^127 - 1] | - | |
FLOAT | 4-byte float | FLOAT | 4-byte float | - | |
DOUBLE | 8-byte float | DOUBLE | 8-byte float | - | |
DECIMAL | DECIMAL(M,D), M Range [1, 65], D Range [0, 30] | DECIMALV3 | DECIMAL(M,D), M Range [1, 38], D Range [0, precision] | - | |
Datetime Type | YEAR | Range: 1901 to 2155 Display Format: YYYY | SMALLINT | Range [-32768, 32767] | - |
TIME | Range: -838:59:59 to 838:59:59 Display Format: hh:mm:ss or hh:mm:ss | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | - | |
DATE | Range: 1000-01-01 to 9999-12-31 Display Format: YYYY-MM-DD | DATEV2 | Range: 0000-01-01 to 9999-12-31 Display Format: YYYY-MM-DD | - | |
DATETIME | 1000-01-01 00:00:00 to 9999-12-31 23:59:59 Display Format: YYYY-MM-DD HH:mm:ss | DATETIMEV2 | 0000-01-01 00:00:00 to 9999-12-31 23:59:59 Print Format: YYYY-MM-dd HH:mm:ss.SSSSSS, time precision is optional. | - | |
TIMESTAMP | UTC 1970-01-01 00:00:01 to 2038-01-19 03:14:07 Display Format: YYYY-MM-DD HH:mm:ss | DATETIMEV2 | Display Format: YYYY-MM-DD HH:mm:ss | TIMESTAMP field data will change with system time zone but DATETIME field data will not, it is recommended to perform timezone conversion according to the business scenario | |
String Type | CHAR | 0 to 255 characters | CHAR | Fixed-length string, range is 1 - 255 | - |
VARCHAR | 0 to 65,535 characters | VARCHAR | Variable-length string, range is 1 - 65533 | If MySQL field length exceeds 65533, it is recommended to convert it to a string | |
TINYTEXT,TEXT | 0 to 255 characters | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | - | |
MEDIUMTEXT,LONGTEXT | 0 to 65535 characters | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | When MySQL field length exceeds 1048576 bytes, there may be precision loss | |
Binary Strings | TINYBLOB,BLOB | Binary string, 0 to 255 bytes | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | - |
MEDIUMBLOB,LONGBLOB | Binary string, 0 to 16,777,215 bytes, maximum 16M | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | When MySQL field length exceeds 1048576 bytes, there may be precision loss | |
BINARY,VARBINARY | Fixed-length binary data, up to 255 bytes | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | - | |
Other | JSON | JSON Data, maximum storage size is 1GB | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | When the MySQL field size exceeds 1 MB, precision may be lost. |
SET,BIT | String collection, up to 64 members | STRING | Variable Length String, maximum (default) support 1048576 bytes (1MB) | - | |
ENUM | Enumeration object, up to 65535 members | UNSUPPORTED | - | Not supported. | |
MySQL to DLC Iceberg Table Creation Field Type Conversion:
MySQL Type (source end) | DLC Iceberg Table (Target End) | Description |
tinyint(1) | int | - |
smallint | smallint | - |
int | int | - |
mediumint | int | - |
bigint | bigint | - |
float | float | - |
double | double | - |
decimal | decimal | - |
datetime | timestamp | - |
timestamp | timestamp | - |
date | date | - |
time | time | - |
tinytext | string | - |
text | string | - |
mediumtext | string | - |
longtext | string | - |
varchar | string | - |
char | string | - |
bool | boolean | - |
tinyblob | binary | - |
mediumblob | binary | - |
blob | binary | - |
longblob | binary | - |
varbinary | binary | - |
binary | binary | - |
decimal unsigned(p,x) | decimal(p+1, x) | Description: decimal(p+1, x)/string (converted to string after exceeding length) |
decimal unsigned(p,x) | decimal(p+1, x) | Description: decimal(p+1, x)/string (converted to string after exceeding length) |
int unsigned | bigint | - |
int unsigned zerofill | bigint | - |
smallint unsigned | int | - |
smallint unsigned zerofill | int | - |
mediumint unsigned | bigint | - |
mediumint unsigned zerofill | bigint | - |
float unsigned | double | - |
float unsigned zerofill | double | - |
double unsigned | decimal(20,0) | - |
double unsigned zerofill | decimal(20,0) | - |
bigint unsigned | decimal(20,0) | - |
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No