tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Cloud WeData
    Documentation
    Download PDF

    Automatic Table Creation Capability of Synchronization Tasks

    Last updated: 2024-11-01 17:33:23
    Download PDF
      DataInLong provides one-click table creation, automatic table creation, and other features in offline and real-time scenarios to help you quickly build target data tables before or during synchronization, thereby improving data link usage efficiency. This article mainly introduces the usage of the corresponding features, supported links, type conversion relationships between core heterogeneous data sources, etc.

      Feature Introduction and Use Cases

      DataInLong offers three table creation methods: one-click table creation, batch table creation, and automatic table creation based on DDL messages:
      Feature Items
      Feature Illustration
      Feature Introduction
      Use cases
      One-Click Table Creation
      
      
      
      This feature can automatically convert multiple table objects specified from the source end to the target end DDL, supporting users to manually modify the converted DDL content and create them in batches at once. Feature Keywords: Heterogeneous DDL Automatic Conversion, Batch Table Creation, Manual Table Creation.
      This feature is suitable for quickly constructing target tables resembling the source table structure for the source end's inventory. For complex table models that require customized business definitions, users can edit the automatically generated DDL statements based on business features to enhance heterogeneous DDL coding efficiency.
      Batch Table Creation
      
      
      
      Automatic Table Creation Based on Source Side DDL Messages
      
      
      
      This feature can monitor in real time whether new table DDL exists at the source end. Once an identifiable new table message is generated at the source end, the target end can automatically generate target table objects based on the source table structure.
      feature keywords: DDL automatic detection and response, automatic table creation.
      This feature is suitable for data synchronization scenarios where it is desired to keep the structure of the target end consistent with the source end in real-time during real-time database migration.

      Supported Links for One-Click Table Creation

      Offline single table tasks and real-time single table tasks support the one-click creation of target tables based on source tables, mainly for scenarios of adding a small number of new database tables to the lake or warehouse.

      Offline Single Table Synchronization Task

      Target Number
      Target
      Supported Source Ends
      1
      Doris
      MySQL,TDSQL-C MySQL,PostgreSQL
      2
      Hive
      MySQL,Oracle,SQL Server,OceanBase,PostgreSQL
      3
      Iceberg
      MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng
      4
      TCHouse-P
      MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng
      5
      DLC
      MySQL,TDSQL-C MySQL,PostgreSQL,Elasticsearch,Kafka

      Real-time Single Table Synchronization Task

      Target Number
      Target
      Supported Source Ends
      1
      TCHouse-P
      MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, SQL Server, PostgreSQL, Oceanbase, Dameng
      2
      Iceberg
      MySQL, TDSQL-C MySQL, TDSQL MySQL, Oracle, PostgreSQL, SQL Server, Oceanbase, Dameng
      3
      Doris
      MySQL,TDSQL-C MySQL,PostgreQL
      4
      DLC
      MySQL,TDSQL-C MySQL,PostgreSQL,Elasticsearch,Kafka

      Supported Links for Batch Table Creation

      Offline batch creation of single table tasks and real-time whole database tasks support bulk creation of target tables based on source tables during the task configuration stage, mainly for scenarios of ingesting a large number of existing database tables into the lake or warehouse.

      Offline Batch Sync Task

      Target Number
      Target
      Supported Source Ends
      1
      Hive
      MySQL,Oracle,SQL Server,PostgreSQL
      2
      Iceberg
      MySQL, TDSQL-C for MySQL, TDSQL for MySQL, Oracle, SQL Server, PostgreSQL, Dameng
      3
      TCHouse-P
      MySQL, TDSQL-C for MySQL, TDSQL for MySQL, Oracle, PostgreSQL, OceanBase, Dameng
      4
      Doris
      MySQL,TDSQL-C MySQL,PostgreSQL
      5
      DLC
      MySQL,TDSQL-C MySQL,PostgreSQL

      Real-time Full Database Migration Task

      Target Number
      Target
      Supported Source Ends
      1
      Doris
      MySQL,TDSQL-C MySQL,TDSQL MySQL
      2
      DLC
      MySQL,TDSQL-C MySQL,TDSQL MySQL

      Support Automatic Table Creation Based on Source Side DDL Messages

      Except for the following links, all other real-time full database links support automatic creation of target databases and tables during task running based on source side DDL messages:
      Serial number
      Source
      Target
      Remarks
      1
      MySQL
      PostgreSQL
      Only supports using existing databases, schemas, and tables
      2
      OceanBase
      Doris,Iceberg
      Only supports using existing databases and tables
      3
      Kafka
      Doris
      Only supports using existing databases and tables

      Core Heterogeneous Data Source Table Creation Type Conversion Relationship

      MySQL to Doris Table Field Type Conversion:

      
      MySQL (source end)
      DORIS (target end)
      Additional Notes
      
      Data Type
      Precision Description
      Data Type
      Precision Description
      
      
      BOOLEAN
      0 represents false, 1 represents true
      BOOLEAN
      0 represents false, 1 represents true
      -
      Value Type
      TINYINT
      Range [-128, 127]
      TINYINT
      Range [-128, 127]
      -
      SMALLINT
      Range [-32768, 32767]
      SMALLINT
      Range [-32768, 32767]
      -
      MEDIUMINT
      Range [-8388608, 8388607]
      INT
      Range [-2147483648, 2147483647]
      -
      INT
      Range [-2147483648, 2147483647]
      INT
      Range [-2147483648, 2147483647]
      -
      BIGINT
      Range [-9223372036854775808, 9223372036854775807]
      BIGINT
      Range [-9223372036854775808, 9223372036854775807]
      -
      UNSIGNED TINYINT
      Range [0, 255]
      SMALLINT
      Range [-32768, 32767]
      -
      UNSIGNED MEDIUMINT
      Range [0, 16777215]
      INT
      Range [-2147483648, 2147483647]
      -
      UNSIGNED INT
      Range [0, 4294967295]
      BIGINT
      Range [-9223372036854775808, 9223372036854775807]
      -
      UNSIGNED BIGINT
      Range [0, 18,446,744,073,709,551,615]
      LARGEINT
      Range [-2^127 + 1 ~ 2^127 - 1]
      -
      FLOAT
      4-byte float
      FLOAT
      4-byte float
      -
      DOUBLE
      8-byte float
      DOUBLE
      8-byte float
      -
      DECIMAL
      DECIMAL(M,D), M Range [1, 65], D Range [0, 30]
      DECIMALV3
      DECIMAL(M,D), M Range [1, 38], D Range [0, precision]
      -
      Datetime Type
      YEAR
      Range: 1901 to 2155 Display Format: YYYY
      SMALLINT
      Range [-32768, 32767]
      -
      TIME
      Range: -838:59:59 to 838:59:59 Display Format: hh:mm:ss or hh:mm:ss
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      -
      DATE
      Range: 1000-01-01 to 9999-12-31 Display Format: YYYY-MM-DD
      DATEV2
      Range: 0000-01-01 to 9999-12-31 Display Format: YYYY-MM-DD
      -
      DATETIME
      1000-01-01 00:00:00 to 9999-12-31 23:59:59 Display Format: YYYY-MM-DD HH:mm:ss
      DATETIMEV2
      0000-01-01 00:00:00 to 9999-12-31 23:59:59 Print Format: YYYY-MM-dd HH:mm:ss.SSSSSS, time precision is optional.
      -
      TIMESTAMP
      UTC 1970-01-01 00:00:01 to 2038-01-19 03:14:07 Display Format: YYYY-MM-DD HH:mm:ss
      DATETIMEV2
      Display Format: YYYY-MM-DD HH:mm:ss
      TIMESTAMP field data will change with system time zone but DATETIME field data will not, it is recommended to perform timezone conversion according to the business scenario
      String Type
      CHAR
      0 to 255 characters
      CHAR
      Fixed-length string, range is 1 - 255
      -
      VARCHAR
      0 to 65,535 characters
      VARCHAR
      Variable-length string, range is 1 - 65533
      If MySQL field length exceeds 65533, it is recommended to convert it to a string
      TINYTEXT,TEXT
      0 to 255 characters
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      -
      MEDIUMTEXT,LONGTEXT
      0 to 65535 characters
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      When MySQL field length exceeds 1048576 bytes, there may be precision loss
      Binary Strings
      TINYBLOB,BLOB
      Binary string, 0 to 255 bytes
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      -
      MEDIUMBLOB,LONGBLOB
      Binary string, 0 to 16,777,215 bytes, maximum 16M
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      When MySQL field length exceeds 1048576 bytes, there may be precision loss
      BINARY,VARBINARY
      Fixed-length binary data, up to 255 bytes
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      -
      Other
      JSON
      JSON Data, maximum storage size is 1GB
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      When the MySQL field size exceeds 1 MB, precision may be lost.
      SET,BIT
      String collection, up to 64 members
      STRING
      Variable Length String, maximum (default) support 1048576 bytes (1MB)
      -
      ENUM
      Enumeration object, up to 65535 members
      UNSUPPORTED
      -
      Not supported.

      MySQL to DLC Iceberg Table Creation Field Type Conversion:

      MySQL Type (source end)
      DLC Iceberg Table (Target End)
      Description
      tinyint(1)
      int
      -
      smallint
      smallint
      -
      int
      int
      -
      mediumint
      int
      -
      bigint
      bigint
      -
      float
      float
      -
      double
      double
      -
      decimal
      decimal
      -
      datetime
      timestamp
      -
      timestamp
      timestamp
      -
      date
      date
      -
      time
      time
      -
      tinytext
      string
      -
      text
      string
      -
      mediumtext
      string
      -
      longtext
      string
      -
      varchar
      string
      -
      char
      string
      -
      bool
      boolean
      -
      tinyblob
      binary
      -
      mediumblob
      binary
      -
      blob
      binary
      -
      longblob
      binary
      -
      varbinary
      binary
      -
      binary
      binary
      -
      decimal unsigned(p,x)
      decimal(p+1, x)
      Description: decimal(p+1, x)/string (converted to string after exceeding length)
      decimal unsigned(p,x)
      decimal(p+1, x)
      Description: decimal(p+1, x)/string (converted to string after exceeding length)
      int unsigned
      bigint
      -
      int unsigned zerofill
      bigint
      -
      smallint unsigned
      int
      -
      smallint unsigned zerofill
      int
      -
      mediumint unsigned
      bigint
      -
      mediumint unsigned zerofill
      bigint
      -
      float unsigned
      double
      -
      float unsigned zerofill
      double
      -
      double unsigned
      decimal(20,0)
      -
      double unsigned zerofill
      decimal(20,0)
      -
      bigint unsigned
      decimal(20,0)
      -
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy