tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Cloud Observability Platform
    Documentation
    Download PDF

    Configuring a DTS Connector

    Last updated: 2024-11-01 20:42:27
    Download PDF

      Overview

      Data Transmission Service (DTS) supports data subscription for MySQL, MariaDB, PostgreSQL, Redis, MongoDB, and other relational databases and NoSQL databases. It gets the data changes of key database businesses and provides them for downstream entities to subscribe to, get, and consume. This facilitates data sync between TencentDB databases and heterogeneous systems, such as cache update, real-time ETL (data warehousing technology) sync, and async business decoupling.
      In EventBridge, you can configure a DTS connector to pull incremental BinLog from the source instance in real time based on DTS data subscription to consume and process logs and deliver them to different targets. This document introduces how to create a DTS connector and the structure of events generated by the DTS connector.

      Feature Description

      For more information, see Data Subscription (for Kafka) Overview.

      Supported databases

      Source Database Type
      Source Database Version
      Subscribable Data Types
      TencentDB for MySQL
      TencentDB for MySQL 5.5.x, 5.6.x, 5.7.x, and 8.0.x
      Data update
      Structure update
      Full instance
      TencentDB for MariaDB
      TencentDB for MariaDB 10.0.10 and 10.1.9
      Data update
      Structure update
      Full instance
      TencentDB for MariaDB (Percona)
      TencentDB for MariaDB (Percona) 5.6.x and 5.7.x
      Data update
      Structure update
      Full instance
      TDSQL for MySQL
      TDSQL for MySQL 5.6.x, 5.7.x, and 8.0.18
      Data update
      Structure update
      Full instance
      TDSQL-C MySQL
      TDSQL-C MySQL 5.7.x and 8.0.x
      Data update
      Structure update
      Full instance
      TDSQL for PostgreSQL
      TDSQL for PostgreSQL
      Data update

      Supported subscription operations

      DTS allows you to subscribe to databases and tables. Specifically, the following three subscription types are supported:
      Data update: Subscription to DML operations.
      Structure update: Subscription to DDL operations.
      Full instance: Subscription to the DML and DDL operations of all tables.

      Use limits

      EventBridge has a limit on the event size. An upstream log greater than 1 MB cannot be successfully delivered to EventBridge for consumption. Pay attention to your log size.
      Under the current scheme, EventBridge can consume data in multiple partitions concurrently and cannot guarantee the consumption sequence.
      To ensure normal data consumption, you need to exercise caution when updating the consumer group account password after creating a connector. If the password is updated, you need to bind the connector again. Otherwise, data consumption may fail.
      DTS supports batch delivery. Batch changes will be delivered in array format in the same event.

      Prerequisites

      1. You have activated the DTS data subscription service and created an instance.
      2. Your sub-account has obtained EventBridge and DTS related operation permissions by using the root account.

      Directions

      1. Log in to the EventBridge console and select Event Bus on the left sidebar.
      2. In the Event Bus list, select the event bus for which you want to configure a DTS connector.
      3. On the event bus details page, click Add in the event connector section, and set the connector type to DTS data subscriptions.
      4. Select the data subscription instance to consume as instructed, and enter information such as the consumer group name, account, and password as needed. If no consumer object is bound, complete related configuration in the DTS console.
      5. Click OK. Then you can view the information of the bound connector on the event bus page.
      6. On the event bus details page, click Manage Event Rules.
      
      
      7. Click Create Event Rule and enter the relevant information as prompted. You need to set Tencent Cloud service to DTS data subscriptions and configure data conversion and complete trigger target binding as needed.
      8. Click OK.

      Event Format

      The format of processed events received by a DTS connector is as follows:

      DDL operation example

       {
         "id":"38cecd93-a9c2-11ec-b952-******d8da53:16",
         "type":"dts:MYSQL:INSERT",
         "specversion":"1.0",
         "source":"dts.cloud.tencent",
         "subject":"cdb-xxxx",
         "time":1648109734,
         "region":"ap-guangzhou",
         "datacontenttype":"application/json;charset=utf-8",
         "tags":null,
         "data":{
            "topic":"topic-subs-xxxx-cdb-xxxx",
            "partition":3,
            "offset":16005,
            "partition_seq":16006,
            "event":{
               "dmlEvent":{
                  "columns":[
                     {
                        "name":"string",
                        "originalType":"text"
                     },
                     {
                        "name":"int",
                        "originalType":"tinyint(4)"
                     },
                     {
                        "name":"time",
                        "originalType":"time"
                     },
                     {
                        "name":"double",
                        "originalType":"double"
                     },
                     {
                        "name":"id",
                        "originalType":"int(11)",
                        "isKey":true
                     },
                     {
                        "name":"float",
                        "originalType":"float"
                     },
                     {
                        "name":"longtext",
                        "originalType":"longtext"
                     }
                  ],
                  "rows":[
                     {
                        "newColumns":[
                           {
                              "dataType":13,
                              "charset":"utf8",
                              "bv":"dG1w"
                           },
                           {
      
                           },
                           {
      
                           },
                           {
                              "dataType":10,
                              "sv":"1"
                           },
                           {
                              "dataType":3,
                              "sv":"3"
                           },
                           {
      
                           },
                           {
      
                           }
                        ]
                     }
                  ]
               }
            },
            "header":{
               "sourceType":1,
               "messageType":2,
               "timestamp":1648109734,
               "serverId":109741,
               "fileName":"mysql-bin.000005",
               "position":2234587,
               "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:16",
               "schemaName":"dts",
               "tableName":"dts_mysql",
               "seqId":16017,
               "isLast":true
            },
            "eb_consumer_time":"2022-03-24 16:15:34.287359965 +0800 CST m=+1120.357657669",
            "eb_connector":""
         }
      }

      DML operation example

      {
         "id":"38cecd93-a9c2-11ec-b952-******8da53:19",
         "type":"dts:MYSQL:DDL",
         "specversion":"1.0",
         "source":"dts.cloud.tencent",
         "subject":"cdb-xxxx",
         "time":1648110060,
         "region":"ap-guangzhou",
         "datacontenttype":"application/json;charset=utf-8",
         "tags":null,
         "data":{
            "topic":"topic-subs-aniwxeewm4-cdb-xxxx",
            "partition":0,
            "offset":16065,
            "partition_seq":16066,
            "event":{
               "ddlEvent":{
                  "schemaName":"dts",
                  "sql":"ALTER TABLE `dts_mysql` ADD COLUMN `t` tinyint (0) NULL , ADD UNIQUE `t` USING btree (`t`)"
               }
            },
            "header":{
               "sourceType":1,
               "messageType":3,
               "timestamp":1648110060,
               "serverId":109741,
               "fileName":"mysql-bin.000005",
               "position":2235430,
               "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:19",
               "seqId":16087,
               "isLast":true
            },
            "eb_consumer_time":"2022-03-24 16:21:01.19682088 +0800 CST m=+1447.267118604",
            "eb_connector":""
         }
      }

      Parameter description

      Parameter
      Description
      id
      Event ID, which is automatically generated by EventBridge. It is the unique ID of an event in EventBridge.
      type
      Event type in three-segment format. For a DTS connector, the format is dts:${Database type}:${Operation type}.
      specversion
      CloudEvents version, which is 1.0 by default. The value of this parameter is automatically generated by EventBridge.
      source
      Event source. For a DTS connector, the value is dts.cloud.tencent.
      subject
      Instance on which an event is generated. For a DTS connector, the value of this parameter is the ID of the database instance bound to the data subscription.
      time
      Time when an event is delivered to EventBridge.
      region
      Region where an event is generated.
      tags
      Resource tags.
      data
      Binlog content of the database.
      data.topic
      Information of the data subscription instance.
      data.partition
      Consumer partition.
      data.offset
      Consumer offset.
      data.event
      There are two types: dmlEvent and ddlEvent. dmlEvent describes the data table schema format and updates. ddlEvent describes specific SQL operations.
      data.header
      Operation log header information, including the database name, table name, and update timestamp.
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy