tencent cloud

All product documents
Event Bridge
DocumentationEvent BridgeEvent SourceConnectorConfiguring a DTS Connector
Configuring a DTS Connector
Download PDF
Last updated: 2024-07-23 15:08:07
Configuring a DTS Connector
Last updated: 2024-07-23 15:08:07
Download PDF

Overview

Data Transmission Service (DTS) supports data subscription for MySQL, MariaDB, PostgreSQL, Redis, MongoDB, and other relational databases and NoSQL databases. It gets the data changes of key database businesses and provides them for downstream entities to subscribe to, get, and consume. This facilitates data sync between TencentDB databases and heterogeneous systems, such as cache update, real-time ETL (data warehousing technology) sync, and async business decoupling.
In EventBridge, you can configure a DTS connector to pull incremental BinLog from the source instance in real time based on DTS data subscription to consume and process logs and deliver them to different targets. This document introduces how to create a DTS connector and the structure of events generated by the DTS connector.

Feature Description

For more information, see Data Subscription (for Kafka) Overview.

Supported databases

Source Database Type
Source Database Version
Subscribable Data Types
TencentDB for MySQL
TencentDB for MySQL 5.5.x, 5.6.x, 5.7.x, and 8.0.x
Data update
Structure update
Full instance
TencentDB for MariaDB
TencentDB for MariaDB 10.0.10 and 10.1.9
Data update
Structure update
Full instance
TencentDB for MariaDB (Percona)
TencentDB for MariaDB (Percona) 5.6.x and 5.7.x
Data update
Structure update
Full instance
TDSQL for MySQL
TDSQL for MySQL 5.6.x, 5.7.x, and 8.0.18
Data update
Structure update
Full instance
TDSQL-C MySQL
TDSQL-C MySQL 5.7.x and 8.0.x
Data update
Structure update
Full instance
TDSQL for PostgreSQL
TDSQL for PostgreSQL
Data update

Supported subscription operations

DTS allows you to subscribe to databases and tables. Specifically, the following three subscription types are supported:
Data update: Subscription to DML operations.
Structure update: Subscription to DDL operations.
Full instance: Subscription to the DML and DDL operations of all tables.

Use limits

EventBridge has a limit on the event size. An upstream log greater than 1 MB cannot be successfully delivered to EventBridge for consumption. Pay attention to your log size.
Under the current scheme, EventBridge can consume data in multiple partitions concurrently and cannot guarantee the consumption sequence.
To ensure normal data consumption, you need to exercise caution when updating the consumer group account password after creating a connector. If the password is updated, you need to bind the connector again. Otherwise, data consumption may fail.
DTS supports batch delivery. Batch changes will be delivered in array format in the same event.

Prerequisites

1. You have activated the DTS data subscription service and created an instance.
2. Your sub-account has obtained EventBridge and DTS related operation permissions by using the root account.

Directions

1. Log in to the EventBridge console and select Event Bus on the left sidebar.
2. In the Event Bus list, select the event bus for which you want to configure a DTS connector.
3. On the event bus details page, click Add in the event connector section, and set the connector type to DTS data subscriptions.
4. Select the data subscription instance to consume as instructed, and enter information such as the consumer group name, account, and password as needed. If no consumer object is bound, complete related configuration in the DTS console.
5. Click OK. Then you can view the information of the bound connector on the event bus page.
6. On the event bus details page, click Manage Event Rules.


7. Click Create Event Rule and enter the relevant information as prompted. You need to set Tencent Cloud service to DTS data subscriptions and configure data conversion and complete trigger target binding as needed.
8. Click OK.

Event Format

The format of processed events received by a DTS connector is as follows:

DDL operation example

 {
   "id":"38cecd93-a9c2-11ec-b952-******d8da53:16",
   "type":"dts:MYSQL:INSERT",
   "specversion":"1.0",
   "source":"dts.cloud.tencent",
   "subject":"cdb-xxxx",
   "time":1648109734,
   "region":"ap-guangzhou",
   "datacontenttype":"application/json;charset=utf-8",
   "tags":null,
   "data":{
      "topic":"topic-subs-xxxx-cdb-xxxx",
      "partition":3,
      "offset":16005,
      "partition_seq":16006,
      "event":{
         "dmlEvent":{
            "columns":[
               {
                  "name":"string",
                  "originalType":"text"
               },
               {
                  "name":"int",
                  "originalType":"tinyint(4)"
               },
               {
                  "name":"time",
                  "originalType":"time"
               },
               {
                  "name":"double",
                  "originalType":"double"
               },
               {
                  "name":"id",
                  "originalType":"int(11)",
                  "isKey":true
               },
               {
                  "name":"float",
                  "originalType":"float"
               },
               {
                  "name":"longtext",
                  "originalType":"longtext"
               }
            ],
            "rows":[
               {
                  "newColumns":[
                     {
                        "dataType":13,
                        "charset":"utf8",
                        "bv":"dG1w"
                     },
                     {

                     },
                     {

                     },
                     {
                        "dataType":10,
                        "sv":"1"
                     },
                     {
                        "dataType":3,
                        "sv":"3"
                     },
                     {

                     },
                     {

                     }
                  ]
               }
            ]
         }
      },
      "header":{
         "sourceType":1,
         "messageType":2,
         "timestamp":1648109734,
         "serverId":109741,
         "fileName":"mysql-bin.000005",
         "position":2234587,
         "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:16",
         "schemaName":"dts",
         "tableName":"dts_mysql",
         "seqId":16017,
         "isLast":true
      },
      "eb_consumer_time":"2022-03-24 16:15:34.287359965 +0800 CST m=+1120.357657669",
      "eb_connector":""
   }
}

DML operation example

{
   "id":"38cecd93-a9c2-11ec-b952-******8da53:19",
   "type":"dts:MYSQL:DDL",
   "specversion":"1.0",
   "source":"dts.cloud.tencent",
   "subject":"cdb-xxxx",
   "time":1648110060,
   "region":"ap-guangzhou",
   "datacontenttype":"application/json;charset=utf-8",
   "tags":null,
   "data":{
      "topic":"topic-subs-aniwxeewm4-cdb-xxxx",
      "partition":0,
      "offset":16065,
      "partition_seq":16066,
      "event":{
         "ddlEvent":{
            "schemaName":"dts",
            "sql":"ALTER TABLE `dts_mysql` ADD COLUMN `t` tinyint (0) NULL , ADD UNIQUE `t` USING btree (`t`)"
         }
      },
      "header":{
         "sourceType":1,
         "messageType":3,
         "timestamp":1648110060,
         "serverId":109741,
         "fileName":"mysql-bin.000005",
         "position":2235430,
         "gtid":"38cecd93-a9c2-11ec-b952-******d8da53:19",
         "seqId":16087,
         "isLast":true
      },
      "eb_consumer_time":"2022-03-24 16:21:01.19682088 +0800 CST m=+1447.267118604",
      "eb_connector":""
   }
}

Parameter description

Parameter
Description
id
Event ID, which is automatically generated by EventBridge. It is the unique ID of an event in EventBridge.
type
Event type in three-segment format. For a DTS connector, the format is dts:${Database type}:${Operation type}.
specversion
CloudEvents version, which is 1.0 by default. The value of this parameter is automatically generated by EventBridge.
source
Event source. For a DTS connector, the value is dts.cloud.tencent.
subject
Instance on which an event is generated. For a DTS connector, the value of this parameter is the ID of the database instance bound to the data subscription.
time
Time when an event is delivered to EventBridge.
region
Region where an event is generated.
tags
Resource tags.
data
Binlog content of the database.
data.topic
Information of the data subscription instance.
data.partition
Consumer partition.
data.offset
Consumer offset.
data.event
There are two types: dmlEvent and ddlEvent. dmlEvent describes the data table schema format and updates. ddlEvent describes specific SQL operations.
data.header
Operation log header information, including the database name, table name, and update timestamp.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon