tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Data Transfer Service
    Documentation
    Download PDF

    Use Instructions

    Last updated: 2024-07-08 17:34:39
    Download PDF
    Category
    Description
    Sync Object
    1. Only sync of base tables and views is supported. Sync of functions, triggers, storage processes, and other objects is not supported.
    2. Only sync of InnoDB database engines is supported. If there are tables with other database engines, an error will be reported during task check.
    3. Interrelated data objects need to be synced at the same time, otherwise sync failure will occur.
    4. In the source TDSQL MySQL, there is a limit on the number of tables, with a maximum number of 5,000 for the entire instance. Exceeding this limit will result in DTS task errors; also, having too many tables will increase the access time at the source, leading to performance fluctuation and decline.
    5. During the incremental sync stage, if source database table names include TDSQLagent and tdsql_sub characters, they might be filtered out or cause sync anomalies, as these table names are the same as TDSQL system's temporary table names. TDSQLagent is a temporary table for scaling, and tdsql_sub tables are subtables for hash-list and hash-range. Therefore, it is recommended not to set the source's tables to be synced with these names.
    Sync Feature
    Currently, the primary key conflict resolution policy only supports conflict overwrite. For both full and incremental stages, primary key data conflicts will be handled by conflict overwrite.
    Source Database Impact
    1. DBbridge occupies certain source database resources while performing full data sync, which may lead to increased load in the source database and add to its stress. If your database configuration is low, it's recommended to perform this during the business off-peak period.
    2. During data sync, DBbridge will use the account that executes the sync task to write into the system database __tencentdb__ in the source database, recording transaction marker ID and other metadata. It's necessary to ensure that the source database has read-write permissions for __tencentdb__.
    To ensure that data comparison issues can be traced, the __tencentdb__ system database will not be deleted from the source database after the sync task is completed
    The space occupied by the __tencentdb__ system database is very small, approximately one thousandth to one ten-thousandth of the source database's storage space (for example, if the source database is 50 GB, then the __tencentdb__ system database is about 5 MB - 50 MB). Moreover, by using a single thread and a waiting connection mechanism, it has almost no impact on the source database's performance and will not compete for resources.
    3. By default, a lock-free sync method is used. The full data export stage does not place a global lock (FTWRL) on the source database; it only places table locks on tables without a primary key.
    Operation Restrictions
    Please do not perform the following operations during sync, as they will cause the sync task to fail.
    1. During the full data export stage, please do not execute any DDL operations that change the database or table structure in the source database.
    2. During the sync task, do not modify or delete user information (including username, password, and permissions) and port numbers in both the source and target databases.
    3. Do not clear Binlogs in the source database.
    Data Type
    1. Modifying the primary key is not supported during incremental sync, including the primary key column, partition table distribution key, comments on primary key columns, adding, deleting, or modifying column fields and their lengths.
    2. Geometry-related data types are not supported, and tasks will report an error when encountering such data types.
    3. For tables with floating point types, the precision inconsistency between full and incremental sync may cause the sync result's precision to be inconsistent.
    4. During the incremental sync process, if the source database generates Binlog statements in the STATEMENT format, it will lead to sync failure.
    5. When the source TDSQL MySQL uses the MariaDB 10.1.x kernel, using the timestamp type without specifying precision (e.g., timestamp(3)) is not supported; otherwise, the DTS task will report an error. To resolve it, remove the precision and then recreate the task.
    Transaction
    1. Scenarios that include both DML and DDL Statements in one transaction are not supported; encountering such a scenario will result in an error.
    2. When the source database is TDSQL MySQL (Kernel MariaDB 5.6), XA Transactions are not supported; encountering XA Transactions will result in an error.
    HA Switch and Scaling
    1. If the source database is a non-GTID Database, DBbridge does not support source HA switchover. Once the source TDSQL MySQL undergoes a switchover, it may cause DBbridge incremental sync interruption.
    2. If the source database is a self-built database, and SET connections are used with TDSQL MySQL, when the sync task starts, and the source adds or deletes a SET node, the DBbridge sync task will report an error. Users are required to modify the SET configuration information in DBbridge (to keep it consistent with the actual SET on the source), and then restart the task. Only then can the information about the newly added or deleted SET be synced.
    Partition Table Sync
    1. The full volume stage supports the sync of primary/secondary partition tables, but the partition syntax needs to comply with the TDSQL MySQL standards. Primary Hash partition tables only support creation through the shardkey method. The key syntax for creating a partition table in TDSQL MySQL is as follows, for detailed syntax please refer to TDSQL MySQL Table Creation Syntax Example.
    Primary Hash partition: shardkey
    Primary Range partition: TDSQL_DISTRIBUTED BY RANGE
    Primary List partition: TDSQL_DISTRIBUTED BY LIST
    Primary Hash partition + secondary Range/List partition: shardkey + PARTITION BY RANGE/LIST
    Primary Range partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY RANGE + PARTITION BY RANGE/LIST
    Primary List partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY LIST + PARTITION BY RANGE/LIST
    2. During the incremental sync stage, it is not supported to intensively create a secondary partition table, then drop a secondary partition table, and then create a secondary partition table again, as this might lead to task exceptions due to table type conflicts. Dropping a non-existent secondary partition table before creating a secondary partition table might cause a deadlock, with no error reported from the task, necessitating manual unlocking.
    3. TDSQL MySQL sync to MySQL/MariaDB/Percona link: If the source database to be synced contains secondary partition tables, they will be single tables on the target side after sync.
    Consistency Check
    1. The scope of data consistency check is limited to comparing the selected database objects in the source database with those synced to the target database. If users write data to the target database during the sync task, this data is not included in the check scope, nor are other advanced objects (such as storage procedures, functions, and views). If Structure Initialization is not selected in the sync task configuration (indicating that table structures are not synced), then table structures are also not checked during the consistency check.
    2. The current check task does not recognize DDL operations. If DDL operations are performed in the source database during sync, the result of the check will be inconsistent. Users need to initiate a new check task to obtain accurate comparison results.
    3. In the consistency check task, the timeout limit for DTS to query data from the source or the target is 10 minutes per query. This applies to each block check query, row check query, etc. If a single query exceeds 10 minutes (for example, when querying large tables from the source), it will cause the check task to fail.
    4. If some DMLs are selected or Where conditions are set for filtering in the sync task configuration, it will cause inconsistencies between the source and target databases. Therefore, consistency checks are not supported. To perform a consistency check, all DMLs and DDLs must be selected.
    5. Data check supports unidirectional sync only; it does not support complex topologies like many-to-one or one-to-many.
    6. The following configuration in a sync task may lead to inconsistent check results. Please be aware of this when creating a check task.
    If the 'Full Data Initialization' option is not selected during data initialization, there may be inconsistencies between the source and the target data, potentially leading to discrepancies in the data check results.
    7. A data consistency check task may increase the load in the source database instance. Therefore, you need to perform such tasks during the business off-peak period.
    8. A data consistency check task can be executed repeatedly, but a DBbridge task can initiate only one consistency check task at any time.
    9. Complete comparison and sampling comparison require the tables to have a primary key or unique key; otherwise, they will be skipped and not checked. Row count check does not require a primary key or unique key.
    10. If the user chooses to end the sync task before the data consistency check is complete, the consistency check will fail.
    11. Consistency checks require the use of the account executing the sync task to create a table CRC_xxx_CMP_xxx in the source database, which is used to record the data comparison information during the sync task.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Contact Us
    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    Submit a Ticket
    7x24 Phone Support
    Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
    Privacy PolicyLegalCookie Policy