tencent cloud

Feedback

Data Sync

Last updated: 2023-03-03 14:41:40

    Feature overview

    The data sync feature refers to real-time data sync between two database sources. It is suitable for cloud-local active-active, multi-site active-active, and cross-border data sync as well as real-time data warehousing.
    Data migration is a one-time short-term task to migrate the entire database and requires a manual cutover operation after migration for business connection to the new database. In contrast, data sync is a continuous task. After a task is created, the data will be continuously synced (almost in real time) to keep consistency between the source and target databases.
    DTS supports the sync of self-built, TencentDB, and third-party cloud databases to TencentDB.
    Cloud-local sync: DTS can sync an IDC-based self-built database to a TencentDB instance and vice versa.
    Cross-cloud sync: DTS can sync a third-party cloud database to TencentDB instance.
    Cross-TencentDB instance sync: DTS can sync TencentDB instances in various scenarios, such as multi-site active-active, cross-border, and cross-Tencent Cloud account sync.

    How it works

    The following takes MySQL sync as an example to describe the data sync feature, where data is exported from the source instance and imported into the target instance, and key steps include structure initialization, full data initialization, and incremental data processing.
    Structure initialization Structure initialization is to create the same table structure information in the target instance as that in the source instance. When configuring a sync task, you can select whether to sync the table structure. If the target database already contains the same structural information, you simply need to sync the data; otherwise, you'll need to sync the information as well.
    Full data initialization After structure initialization is completed, DTS will initialize the existing data, i.e., exporting all existing data from the source instance and import it into the target instance.
    Incremental data processing In incremental data processing, DTS gets the incremental data continuously through the source instance's binlog and persistently stores such data in the intermediate storage after a series filtering and conversion operations. After importing the full data, it continuously replays the changed incremental data stored in the intermediate storage into the target database in order to ensure data consistency between the source and target databases.
    

    Data conflict resolution

    During a DTS sync task, the data in the source and target databases may have conflicts. DTS can check for duplicate table name and primary key conflicts.
    Duplicate table name conflict
    If the source and target databases have tables with the same name, the task can report an error (Precheck and report error) or append the data in the source database to the table with the same name in the target database (Ignore and execute).
    Primary key conflict
    DTS can check for primary key conflicts and provides the following conflict resolution methods. For specific implementation examples, see Selecting Data Sync Conflict Resolution Policy.
    Report: During a sync task, if an INSERT statement in the source database has a primary key conflict with the data in the target database, the task will report an error and pause. You need to handle the conflict manually first before proceeding.
    Ignore: During a sync task, if an INSERT statement in the source database has a primary key conflict with the data in the target database, the data inserted into the source database will be ignored, and the data in the target database will prevail.
    Overwrite: During a sync task, if an INSERT or UPDATE statement has a primary key conflict with the data in the target database, the data in the target database will be overwritten by the inserted or updated data in the source database.

    Supported topologies

    The basic unit of the sync service is one-way sync. During configuration, you can choose to use data definition language (DDL) or data manipulation language (DML) for sync. By combining one-way sync tasks, you can customize various complex topologies. In a complex topology, technical measures will be used for DML operations to avoid data loop. However, for DDL, the data sync service will check for loop during configuration to avoid forming DDL loop.
    Below are some common topologies, which you can customize by purchasing multiple sync instances. For detailed directions on how to create a complex topology, see Creating Two-Way Sync Data Structure, Creating Many-to-One Sync Data Structure, or Creating Multi-Site Active-Active IDC Architecture.
    One-to-one one-way sync
    
    Cascaded one-way sync
    
    One-to-many one-way sync
    
    Many-to-one one-way sync
    
    Two-way sync
    
    Cascaded two-way sync
    

    Typical use cases

    By using DTS, you can sync data between MySQL databases in multiple regions to implement multi-site active-active deployment. Database instances in each region can run in the cloud or your self-built IDC.
    

    Restrictions

    You can implement two-way sync by combining one-way sync tasks, but there are the following restrictions:
    Your business should cooperate in terms of data consistency and cannot update the data record with the same primary key on two nodes; otherwise, a primary key conflict or mutual overwrite may occur. For example, you can choose to update the data records with primary keys 1, 3, and 5 on node A and data records with primary keys 2, 4, and 6 on node B.
    If a data sync conflict occurs, DTS will process the data strictly according to the selected conflict resolution policy. You need to confirm whether the corresponding policy meets your business expectation during configuration.
    DML statements support two-way sync, but DDL statements support only one-way sync. To create two-way sync, ensure that DDL sync is disabled in one of the one-way instances.

    Supported database types

    For more information on the source and target database types, versions, and sync types supported by data sync, see Databases Supported by Data Sync.

    Supported advanced features

    Feature
    Description
    Documentation
    Two-way sync, ring sync, and many-to-one sync
    Complex sync topologies such as two-way sync, ring sync, and many-to-one sync are supported.
    Cross-account sync
    Data sync between instances under different Tencent Cloud accounts is supported.
    Cross-version sync of most databases
    The target database version should be equal to or later than the source database version; for example, data on v5.5.x can be synced to v5.5.x, v5.6.x, or later. The last digit in the version number is the minor version number, which is not restricted.
    -
    Table conflict check
    The duplicate table name check policy is provided.
    -
    Primary key conflict check
    The following three primary key conflict resolution policies are supported:
    Report: If a primary key conflict of tables is found during data sync, the system will report an error and pause the data sync task.
    Ignore: If a primary key conflict is found during data sync, the system will keep the primary key record in the target database.
    Overwrite: If a primary key conflict is found during data sync, the system will use the primary key record in the source database to overwrite that in the target database.
    DML and DDL filtering
    You can select the data types to be synced, including INSERT, UPDATE, and DELETE.
    You can select the specific DDL operation, such as CREATE TABLE, CREATE VIEW, and DROP INDEX.
    WHERE conditional filter
    You can customize a filter for a single table.
    Sync of views and advanced objects
    Views, procedures, and functions can be synced.
    -
    Task progress visualization
    Information such as sync steps and progress can be displayed.
    -
    Metric monitoring and default alarm policy
    Data sync metrics can be monitored.
    Default configuration is supported for data sync event monitoring to automatically notify you of abnormal events.
    Instance restart or upgrade
    During incremental data sync, the source and target instances can be restarted or upgraded.
    -
    HA switch
    HA switch of the source instance is supported if GTID is enabled.
    HA switch of the target database is supported.
    -
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support