DataInLong provides the ability to read and write TDSQL for MySQL. This document introduces the pre-environment configuration for real-time data synchronization using TDSQL-C MySQL and the current capability support.
Supported Versions
Currently, DataInLong supports TDSQL-C MySQL single table and database-level real-time reading and single table writing. To use real-time synchronization capabilities, the following version limitations must be observed:
Engine types
Edition
MySQL
Only supports the original InnoDB type and does not support the TDStore type
Use Limits
Dynamic shard addition is not supported. You need to pause the task before adding shards.
Binlog logs must be enabled, and only the TDSQL MySQL server Binlog configuration format set to ROW is supported.
Tables without primary keys may have data duplication since exactly once cannot be guaranteed. Therefore, it is best to ensure the table has a primary key for real-time sync tasks.
XA ROLLBACK is not supported. Real-time synchronization tasks will not perform rollback operations on data prepared by XA PREPARE. If you need to handle XA ROLLBACK scenarios, you need to manually remove the XA ROLLBACK tables from the real-time synchronization task, re-add the tables, and then synchronize again.
Set TDSQL MySQL session timeout:
When creating an initial consistent snapshot for a large database, the connection you establish may time out when reading the table. You can prevent this behavior by configuring interactive_timeout and wait_timeout in the TDSQL MySQL configuration file.
interactive_timeout: The number of seconds the server waits for activity on an interactive connection before closing it. See documentation .
wait_timeout: The number of seconds the server waits for activity on a non-interactive connection before closing it. See documentation .
Database source configuration
Data source settings
Parameter
Description
Data Source
Select the TDSQL MySQL data source that needs to be synchronized.
Source Table
All databases and tables: Monitor all databases under the data source. Newly added databases and tables during the task run will be synchronized to the target by default.
Specific table: Under this option, you need to specify the exact table name. After setting, the task will only synchronize the specified table; if you need to add a new table for synchronization, you need to stop and restart the task.
Specific database: Under this option, you need to specify the exact database name, using regular expression for table names. After setting, new tables that match the table name expression during the task run will be synchronized to the target by default.
Read Mode
Full + Increment: Data synchronization is divided into full and increment phases. After the full phase is completed, the task enters the increment phase. The full phase will synchronize historical data in the database, and the incremental phase starts synchronizing from the binlog cdc location after the task starts.
Increment Only: Only synchronize data from the binlog cdc position after the task starts.
Consistency Semantics
Only represents the consistency semantics of the reading end. Supports At-least-once and Exactly-once.
At-least-once: Data may be read more than once, relying on the destination end's deduplication to ensure data consistency. Suitable for scenarios with large data volumes in the full phase and using non-numeric primary keys, with high synchronization performance requirements.
Exactly-once: Data is read strictly once, with some performance losses. Tables without a primary key and unique index column are not supported. Suitable for general scenarios where the source table has a numeric primary key or unique index column.
The current version's two modes are state-incompatible. If the mode is changed after task submission, stateful restart is not supported.
Filter Operation
Supports three operations: insert, update, and delete. Data of the specified operation type will not be synchronized when set.
Time Zone
Set the timezone for log time, default is Shanghai.
Advanced Settings (optional)
You can configure parameters according to business needs.
Real-time Single Table Read Node Configuration
Parameter
Description
Data Source
Select the data source where the tables to be synced are located
Database
Select the database where the tables to be synced are located
Table
Support selection or manual entry of the table name to be written to.
Shard Column
Shard columns are used to split the table into multiple shards for synchronization.
For tables with a primary key, it is recommended to prioritize the table's primary key as the shard column.
For tables without a primary key, it is recommended to select a column with an index as the shard column and ensure that there are no data update operations on the shard column; otherwise, only At-Least-Once Semantics can be guaranteed.
Read Mode
Full + Increment: Data synchronization is divided into full and increment phases. After the full phase is completed, the task enters the increment phase. The full phase will synchronize historical data in the database, and the incremental phase starts synchronizing from the binlog cdc location after the task starts.
Increment Only: Only synchronize data from the binlog cdc position after the task starts.
Consistency Semantics
Only represents the consistency semantics of the reading end. Supports At-least-once and Exactly-once.
At-least-once: Data may be read more than once, relying on the destination end's deduplication to ensure data consistency. Suitable for scenarios with large data volumes in the full phase and using non-numeric primary keys, with high synchronization performance requirements.
Exactly-once: Data is read strictly once, with some performance losses. Tables without a primary key and unique index column are not supported. Suitable for general scenarios where the source table has a numeric primary key or unique index column.
The current version's two modes are state-incompatible. If the mode is changed after task submission, stateful restart is not supported.
Filter Operation
After setting, data of the specified operation types will be excluded from synchronization. Supports insert, update, and delete.
Time Zone
Set the timezone for log time, default is Shanghai.
Advanced Settings (Optional)
You can configure parameters according to business needs.
Real-time Single Table Write Node Configuration
Parameter
Description
Data Destination
TDSQL MySQL Data Source for writing.
Database
Supports selection or manual input of the database name to write to
By default, the database bound to the data source is used as the default database. Other databases need to be manually entered.
If the data source network is not connected and the database information cannot be fetched directly, you can manually enter the database name. Data synchronization can still be performed when the Data Integration network is connected.
Table
Support selection or manual entry of the table name to be written to.
If the data source network is not connected and the table information cannot be fetched directly, you can manually enter the table name. Data synchronization can still be performed when the Data Integration network is connected.
Advanced Settings (Optional)
You can configure parameters according to business needs.
Data type conversion support
Read
Field Type
Supported
Internal Mapping Fields
Remarks
TINYINT
Yes
TINYINT
TINYINT(1) is mapped to BOOLEAN by default
A single table task can map TINYINT(1) to tinyint through advanced settings
SMALLINT
Yes
SMALLINT
-
TINYINT_UNSIGNED
Yes
SMALLINT
-
TINYINT_UNSIGNED_ZEROFILL
Yes
SMALLINT
-
INT
Yes
INT
-
INTEGER
Yes
INT
-
YEAR
Yes
INT
-
MEDIUMINT
Yes
INT
-
SMALLINT_UNSIGNED
Yes
INT
-
SMALLINT_UNSIGNED_ZEROFILL
Yes
INT
-
BIGINT
Yes
LONG
-
INT_UNSIGNED
Yes
LONG
-
MEDIUMINT_UNSIGNED
Yes
LONG
-
MEDIUMINT_UNSIGNED_ZEROFILL
Yes
LONG
-
INT_UNSIGNED_ZEROFILL
Yes
LONG
-
BIGINT_UNSIGNED
Yes
DECIMAL
DECIMAL(20,0)
BIGINT_UNSIGNED_ZEROFILL
Yes
DECIMAL
DECIMAL(20,0)
SERIAL
Yes
DECIMAL
DECIMAL(20,0)
FLOAT
Yes
FLOAT
-
FLOAT_UNSIGNED
Yes
FLOAT
-
FLOAT_UNSIGNED_ZEROFILL
Yes
FLOAT
-
DOUBLE
Yes
DOUBLE
-
DOUBLE_UNSIGNED
Yes
DOUBLE
-
DOUBLE_UNSIGNED_ZEROFILL
Yes
DOUBLE
-
DOUBLE_PRECISION
Yes
DOUBLE
-
DOUBLE_PRECISION_UNSIGNED
Yes
DOUBLE
-
ZEROFILL
Yes
DOUBLE
-
REAL
Yes
DOUBLE
-
REAL_UNSIGNED
Yes
DOUBLE
-
REAL_UNSIGNED_ZEROFILL
Yes
DOUBLE
-
NUMERIC
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
NUMERIC_UNSIGNED
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
NUMERIC_UNSIGNED_ZEROFILL
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
DECIMAL
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
DECIMAL_UNSIGNED
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
DECIMAL_UNSIGNED_ZEROFILL
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
FIXED
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
FIXED_UNSIGNED
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
FIXED_UNSIGNED_ZEROFILL
Yes
DECIMAL
Using the actual precision of the user database p<=38 maps to DECIMAL, 38 < p <= 65 maps to String
