Tencent Cloud

Recent Pages

Use Instructions

Last updated: 2024-07-08 19:02:56

Category
Description
Sync Object
1. Only sync of base tables and views is supported. Sync of functions, triggers, storage processes, and other objects is not supported.
2. Only sync of InnoDB, MyISAM, and TokuDB database engines is supported. Tables with any other data engines will be skipped by default.
3. Interrelated data objects need to be synced at the same time, otherwise sync failure will occur.
4. During the incremental sync stage, if source database table names include TDSQLagent and tdsql_sub characters, they might be filtered out or cause sync anomalies, as these table names are the same as TDSQL system's temporary table names. TDSQLagent is a temporary table for scaling, and tdsql_sub tables are subtables for hash-list and hash-range. Therefore, it is recommended not to set the source's tables to be synced with these names.
Sync Feature
1. A single task can associate with multiple Topics, but the same Topic cannot be used by multiple sync tasks simultaneously, as it would lead to data corruption. The data consumed would be from different tasks, and could also lead to exceptions in data processing or errors during task retries.
2. During the full sync stage, for every 100,000 rows of data synced, a checkpoint message is inserted into the target Kafka to mark the current data sync position.
3. The single message synced to the target Kafka by DBbridge has a performance limit. It is recommended that single row data in the source database not exceed 8 MB to avoid errors during the sync process.
4. If a user is sure to rename a table during the sync process, when delivered to the target Kafka, the partitioning rule will match the Topic and partition according to the new table name.
Source Database Impact
1. During data sync, DBbridge will use the account that executes the sync task to write the system database __tencentdb__ in the source database, recording transaction marker ID and other metadata. It is necessary to ensure that the source database has read-write permissions for __tencentdb__.
To ensure that data comparison issues can be traced, the __tencentdb__ system database will not be deleted from the source database after the sync task is completed
The space occupied by the __tencentdb__ system database is very small, approximately one thousandth to one ten-thousandth of the source database's storage space (for example, if the source database is 50 GB, then the __tencentdb__ system database is about 5 MB - 50 MB). Moreover, by using a single thread and a waiting connection mechanism, it has almost no impact on the source database's performance and will not compete for resources.
2. By default, a lock-free sync method is used. The full data export stage does not place a global lock (FTWRL) on the source database; it only places table locks on tables without a primary key.
Target Kafka requirements
1. It is necessary to modify the message retention time and message size limit in the target Kafka.
The message retention time is recommended to be set for 3 days. Data exceeding the retention time will be cleared. Users are advised to consume the data in a timely manner within the set time. The message size limit refers to the maximum size of a single message's memory that Kafka can receive. When setting, it needs to be larger than the maximum size of a single row of data in the source database table, to ensure that data from the source database can be properly delivered to Kafka.
2. It is recommended that the target Topic is empty. Also, during the sync task process, do not write data into the Topic selected for sync at the target, as it may cause message disorder and task errors.
Restart impact
During the sync task, if a task restart occurs (e.g., a restart due to HA switchover at the source database or a restart caused by back-end task anomaly detection), it may lead to duplicated data in the target Kafka.
DBbridge syncs based on the smallest data cell (in the full volume phase, a chunk of data from a single table object is the smallest data cell; in the incremental phase, each marked location is a data cell). If a data cell has finished syncing and then a restart occurs, it will not cause data duplication; however, if a restart occurs while a data cell is still syncing, the data cell will need to be resynced upon restart, leading to data duplication.
If users are particularly concerned about duplicated data, please set deduplication logic when consuming data.
 Operation Restrictions
1. During the full data export stage, please do not execute any DDL operations that change the database or table structure in the source database.
2. Do not modify or delete user information (including username, password, and permissions) and port numbers in the source database and the target.
3. Do not clear Binlogs in the source database.
Data Type
1. When TDSQL MySQL is used as the source database and connected via the proxy method, if the source database employs the float data type for floating numbers, it will cause data precision issues in the full volume phase. To maintain data precision, it is recommended to use the double data type. The impact of precision issues in the full volume phase is as follows:
The precision of data sync between the full volume stage and the incremental stage is inconsistent.
Using float as the key value, the primary key data operated in the full volume stage and the incremental stage are inconsistent.
2. During the full data sync stage, DBbridge exports data from the source database and imports it into the target Kafka, both using the utf8mb4 character set to avoid garbled character issues due to unrecognized character sets.
3. Geometry-related data types are not supported, and tasks will report an error when encountering such data types.
4. During the incremental sync process, if the source database generates Binlog statements in the STATEMENT format, it will lead to sync failure.
5. When the data type delivered to Kafka is selected as OGG JSON (ROW), the source database's tables cannot contain column names like gtid, current_ts, op_ts, op_type, pos, primary_keys, and set_id, table. These column names conflict with the field names in the parsed JSON header, leading to data being overwritten and abnormal consumption data if these column names exist in the source database's tables.
6. Before the sync task, for the full volume of data already existing in the source database, since there's no accurate time of data insertion, the happenAt field in the Consumer Demo will display as 1970-01-01 08:00:00, which needs no attention. When the incremental data is consumed, the time field can be correctly displayed.
Transaction
Scenarios that include both DML and DDL statements in one transaction are not supported; encountering such a scenario will result in an error.
HA Switch and Scaling
1. If the source database is a non-GTID database, DBbridge does not support source HA switchover. Once the source TDSQL MySQL undergoes a switchover, it may cause DBbridge incremental sync interruption.
2. DBbridge uses SET for direct connection to TDSQL MySQL, and does not support scaling of TDSQL MySQL. When Proxy is used to connect to TDSQL MySQL, logical scaling and scale-out of TDSQL MySQL are supported. However, DBbridge may report errors during scale-out.
Partition Table Sync
1. The syntax of primary/secondary partition tables must comply with standards. Primary Hash partition tables only support creation through the shardkey method.
The key syntax for creating partition tables in TDSQL MySQL is as follows. For detailed syntax, refer to TDSQL MySQL Create Table Syntax Example.
Primary Hash partition: shardkey
Primary Range partition: TDSQL_DISTRIBUTED BY RANGE
Primary List partition: TDSQL_DISTRIBUTED BY LIST
Primary Hash partition + secondary Range/List partition: shardkey + PARTITION BY RANGE/LIST
Primary Range partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY RANGE + PARTITION BY RANGE/LIST
Primary List partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY LIST + PARTITION BY RANGE/LIST
2. During the incremental sync stage, concurrent DDL operations are not supported. The next DDL operation should be executed after the previous one takes effect, otherwise errors may occur due to out-of-order DDL operations. Also, rapid create, drop and create operations on tables with the same name are not supported, as they may cause incorrect table type errors.
Designated Position Sync
1. If full sync and incremental sync are executed separately, please note that during full sync, there cannot be DDL changes between the location of the fully synced data by DBbridge (position 1) and the start location (position 2) set for incremental sync; otherwise, the task errors will occur.
2. From initiating synced data to before the start of the incremental task (i.e., before the task step changes from finding the specified location to syncing the increment), it is recommended not to carry out master-slave switch, increase sharding, or redo standby machine operations on the source library, as these may affect DBbridge's access to the GTID location of the source library. Also, the source cannot perform DDL operations; otherwise, the task errors will occur.
3. Because designated position sync is determined based on the time in the Binlog context (SET TIMESTAMP = XXXX) to judge its GTID, to ensure the accuracy of synced data, it is advised not to modify this context.
4. Please ensure that the database's setting time zone is consistent with the current console time zone (i.e., the browser's time zone), or convert it to the corresponding time of the database setting time zone; otherwise, the results of the designated position sync may not meet expectations.
5. Please make sure that the time is consistent across all cluster nodes of TDSQL MySQL; otherwise, the results of the designated position sync may not meet expectations.
6. If the first event at the specified sync point is a two-stage XA transaction, then that XA transaction will not be synced.
﻿

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support

tencent cloud

Recent Pages

Use Instructions

Was this page helpful?

Was this page helpful?

Category	Description
Sync Object	1. Only sync of base tables and views is supported. Sync of functions, triggers, storage processes, and other objects is not supported. 2. Only sync of InnoDB, MyISAM, and TokuDB database engines is supported. Tables with any other data engines will be skipped by default. 3. Interrelated data objects need to be synced at the same time, otherwise sync failure will occur. 4. During the incremental sync stage, if source database table names include TDSQLagent and tdsql_sub characters, they might be filtered out or cause sync anomalies, as these table names are the same as TDSQL system's temporary table names. TDSQLagent is a temporary table for scaling, and tdsql_sub tables are subtables for hash-list and hash-range. Therefore, it is recommended not to set the source's tables to be synced with these names.
Sync Feature	1. A single task can associate with multiple Topics, but the same Topic cannot be used by multiple sync tasks simultaneously, as it would lead to data corruption. The data consumed would be from different tasks, and could also lead to exceptions in data processing or errors during task retries. 2. During the full sync stage, for every 100,000 rows of data synced, a checkpoint message is inserted into the target Kafka to mark the current data sync position. 3. The single message synced to the target Kafka by DBbridge has a performance limit. It is recommended that single row data in the source database not exceed 8 MB to avoid errors during the sync process. 4. If a user is sure to rename a table during the sync process, when delivered to the target Kafka, the partitioning rule will match the Topic and partition according to the new table name.
Source Database Impact	1. During data sync, DBbridge will use the account that executes the sync task to write the system database `__tencentdb__` in the source database, recording transaction marker ID and other metadata. It is necessary to ensure that the source database has read-write permissions for `__tencentdb__`. To ensure that data comparison issues can be traced, the `__tencentdb__` system database will not be deleted from the source database after the sync task is completed The space occupied by the `__tencentdb__` system database is very small, approximately one thousandth to one ten-thousandth of the source database's storage space (for example, if the source database is 50 GB, then the `__tencentdb__` system database is about 5 MB - 50 MB). Moreover, by using a single thread and a waiting connection mechanism, it has almost no impact on the source database's performance and will not compete for resources. 2. By default, a lock-free sync method is used. The full data export stage does not place a global lock (FTWRL) on the source database; it only places table locks on tables without a primary key.
Target Kafka requirements	1. It is necessary to modify the message retention time and message size limit in the target Kafka. The message retention time is recommended to be set for 3 days. Data exceeding the retention time will be cleared. Users are advised to consume the data in a timely manner within the set time. The message size limit refers to the maximum size of a single message's memory that Kafka can receive. When setting, it needs to be larger than the maximum size of a single row of data in the source database table, to ensure that data from the source database can be properly delivered to Kafka. 2. It is recommended that the target Topic is empty. Also, during the sync task process, do not write data into the Topic selected for sync at the target, as it may cause message disorder and task errors.
Restart impact	During the sync task, if a task restart occurs (e.g., a restart due to HA switchover at the source database or a restart caused by back-end task anomaly detection), it may lead to duplicated data in the target Kafka. DBbridge syncs based on the smallest data cell (in the full volume phase, a chunk of data from a single table object is the smallest data cell; in the incremental phase, each marked location is a data cell). If a data cell has finished syncing and then a restart occurs, it will not cause data duplication; however, if a restart occurs while a data cell is still syncing, the data cell will need to be resynced upon restart, leading to data duplication. If users are particularly concerned about duplicated data, please set deduplication logic when consuming data.
Operation Restrictions	1. During the full data export stage, please do not execute any DDL operations that change the database or table structure in the source database. 2. Do not modify or delete user information (including username, password, and permissions) and port numbers in the source database and the target. 3. Do not clear Binlogs in the source database.
Data Type	1. When TDSQL MySQL is used as the source database and connected via the proxy method, if the source database employs the float data type for floating numbers, it will cause data precision issues in the full volume phase. To maintain data precision, it is recommended to use the double data type. The impact of precision issues in the full volume phase is as follows: The precision of data sync between the full volume stage and the incremental stage is inconsistent. Using float as the key value, the primary key data operated in the full volume stage and the incremental stage are inconsistent. 2. During the full data sync stage, DBbridge exports data from the source database and imports it into the target Kafka, both using the utf8mb4 character set to avoid garbled character issues due to unrecognized character sets. 3. Geometry-related data types are not supported, and tasks will report an error when encountering such data types. 4. During the incremental sync process, if the source database generates Binlog statements in the STATEMENT format, it will lead to sync failure. 5. When the data type delivered to Kafka is selected as OGG JSON (ROW), the source database's tables cannot contain column names like gtid, current_ts, op_ts, op_type, pos, primary_keys, and set_id, table. These column names conflict with the field names in the parsed JSON header, leading to data being overwritten and abnormal consumption data if these column names exist in the source database's tables. 6. Before the sync task, for the full volume of data already existing in the source database, since there's no accurate time of data insertion, the happenAt field in the Consumer Demo will display as 1970-01-01 08:00:00, which needs no attention. When the incremental data is consumed, the time field can be correctly displayed.
Transaction	Scenarios that include both DML and DDL statements in one transaction are not supported; encountering such a scenario will result in an error.
HA Switch and Scaling	1. If the source database is a non-GTID database, DBbridge does not support source HA switchover. Once the source TDSQL MySQL undergoes a switchover, it may cause DBbridge incremental sync interruption. 2. DBbridge uses SET for direct connection to TDSQL MySQL, and does not support scaling of TDSQL MySQL. When Proxy is used to connect to TDSQL MySQL, logical scaling and scale-out of TDSQL MySQL are supported. However, DBbridge may report errors during scale-out.
Partition Table Sync	1. The syntax of primary/secondary partition tables must comply with standards. Primary Hash partition tables only support creation through the shardkey method. The key syntax for creating partition tables in TDSQL MySQL is as follows. For detailed syntax, refer to TDSQL MySQL Create Table Syntax Example. Primary Hash partition: shardkey Primary Range partition: TDSQL_DISTRIBUTED BY RANGE Primary List partition: TDSQL_DISTRIBUTED BY LIST Primary Hash partition + secondary Range/List partition: shardkey + PARTITION BY RANGE/LIST Primary Range partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY RANGE + PARTITION BY RANGE/LIST Primary List partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY LIST + PARTITION BY RANGE/LIST 2. During the incremental sync stage, concurrent DDL operations are not supported. The next DDL operation should be executed after the previous one takes effect, otherwise errors may occur due to out-of-order DDL operations. Also, rapid create, drop and create operations on tables with the same name are not supported, as they may cause incorrect table type errors.
Designated Position Sync	1. If full sync and incremental sync are executed separately, please note that during full sync, there cannot be DDL changes between the location of the fully synced data by DBbridge (position 1) and the start location (position 2) set for incremental sync; otherwise, the task errors will occur. 2. From initiating synced data to before the start of the incremental task (i.e., before the task step changes from finding the specified location to syncing the increment), it is recommended not to carry out master-slave switch, increase sharding, or redo standby machine operations on the source library, as these may affect DBbridge's access to the GTID location of the source library. Also, the source cannot perform DDL operations; otherwise, the task errors will occur. 3. Because designated position sync is determined based on the time in the Binlog context (SET TIMESTAMP = XXXX) to judge its GTID, to ensure the accuracy of synced data, it is advised not to modify this context. 4. Please ensure that the database's setting time zone is consistent with the current console time zone (i.e., the browser's time zone), or convert it to the corresponding time of the database setting time zone; otherwise, the results of the designated position sync may not meet expectations. 5. Please make sure that the time is consistent across all cluster nodes of TDSQL MySQL; otherwise, the results of the designated position sync may not meet expectations. 6. If the first event at the specified sync point is a two-stage XA transaction, then that XA transaction will not be synced.

tencent cloud

Sign Up

Log in

Recent Pages

Use Instructions

Was this page helpful?

Was this page helpful?