tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Data Transfer Service
    Documentation
    Download PDF

    Creating MariaDB/Percona Data Subscription

    Last updated: 2024-07-08 16:52:02
    Download PDF
      This document describes how to create a data subscription task in DTS for MariaDB and Percona. MariaDB is used as an example, and the related operations also apply to Percona.

      Prerequisites

      You have prepared a source database to be subscribed to, and the database version meets the requirements. For more information, see Databases Supported by Data Subscription.
      You have enabled the binlog in the source database.
      You have created a subscription account in the source database and granted the account the following permissions: REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT for all objects. Authorization statements are as follows:
      CREATE USER 'account' IDENTIFIED BY 'password';
      GRANT SELECT, REPLICATION CLIENT,REPLICATION SLAVE,PROCESS ON *.* TO 'account'@'%';
      FLUSH PRIVILEGES;

      Restrictions

      Currently, the subscribed message content is retained for one day by default. The data will be cleared when the retention period ends. Therefore, you need to consume the data promptly.
      The region where the data is consumed should be the same as that of the subscribed instance.
      Geometry data types are not supported currently.
      During the process of subscribing to a task, if you perform operations such as modifying the subscription object, the task will be restarted, which may cause duplicated data consumption on the Kafka client.
      DTS transfers data based on the smallest data unit. Every time the incremental data is marked with a checkpoint, it will be considered a data unit. If the transfer of one data unit is completed before the task restart, it will not cause data duplication; if one data unit is still being transferred during the task restart, the data unit needs to be pulled again after the restart to ensure data integrity, which will lead to data duplication.
      If you are more concerned about duplicate data, you can implement deduplication logic when consuming data.

      Subscribable SQL Operations

      Operation Type
      Supported SQL Operations
      DML
      INSERT, UPDATE, DELETE
      DDL
      CREATE DATABASE, DROP DATABASE, CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE

      Directions

      1. Log in to the DTS console, select Data Subscription on the left sidebar, and click Create Subscription.
      2. On the Create Subscription page, select the corresponding configuration and click Buy Now.
      Billing Mode: Monthly subscription and pay-as-you-go billing are supported.
      Region: If the source database is a TencentDB database, select the region of the source database instance here. Otherwise, select the region where data is consumed. If there is no special requirement for the data consumption region, select a region closest to the source database.
      Database: Select the database type.
      Edition: Select Kafka Edition. You can directly consume data on a Kafka client.
      Subscribed Instance Name: Edit the name of the currently subscribed instance.
      3. After successful purchase, return to the data subscription list. You need to click Configure Subscription in the Operation column to configure the newly purchased subscription before you can use it.
      4. On the subscription configuration page, select the source database information, click Test Connectivity, and click Next after the test is passed.
      Configuration Item
      Description
      Instance Type
      The database type selected during purchase.
      We recommend that you subscribe to the primary database because the subscription service only puts a little pressure on the source database.
      If the source database is TencentDB for MariaDB, you can select a disaster recovery or read-only instance.
      Access Type
      Select a type based on your scenario. For the preparations for different access types, see Preparations > Overview.
      Public Network: The source database can be accessed through a public IP.
      Self-Build on CVM: The source database is deployed on a CVM instance. For more information on CVM, see Cloud Virtual Machine (CVM).
      Direct Connect: The source database can be interconnected with VPCs through Direct Connect. For more information on Direct Connect, see Direct Connect.
      VPN Access: The source database can be interconnected with VPCs through VPN Connections. For more information on VPN Connections, see VPN Connections.
      Database: The source database is a TencentDB database.
      CCN: The source database can be interconnected with VPCs through CCN. For more information on CCN, see Cloud Connect Network (CCN).
      Public Network
      Host Address: IP address or domain name of the source database.
      Port: Port used by the source database.
      Self-Build on CVM
      CVM Instance: The ID of the CVM instance.
      Port: Port used by the source database.
      Direct Connect
      VPC-Based Direct Connect Gateway: Only VPC-based direct connect gateway is supported. Confirm the network type associated with the gateway.
      VPC: Select a VPC and subnet associated with the VPC-based Direct Connect gateway or VPN gateway.
      Host Address: IP address of the source database.
      Port: Port used by the source database.
      VPN Access
      VPN Gateway: Select a VPN gateway instance.
      VPC: Select a VPC and subnet associated with the VPC-based Direct Connect gateway or VPN gateway.
      Host Address: IP address of the source database.
      Port: Port used by the source database.
      Database
      Instance Name: The ID of the source database instance.
      CCN
      Host Address: IP address of the source database server.
      Port: Port used by the source database.
      VPC-Based CCN Instance: The name of the CCN instance.
      Accessed VPC: It refers to the VPC in CCN over which the subscription link is connected. You need to select a VPC other than the VPC to which the source database belongs. For example, if the database in Guangzhou is used as the source database, select another region, such VPC-Chengdu or VPC-Shanghai, as the accessed VPC.
      Subnet: Name of the subnet of the selected VPC.
      Region of Accessed VPC: The region of the source database selected during task purchase must be the same as the region of the accessed VPC; otherwise, DTS will change the former to the latter.
      Account/Password
      Account/Password: Enter the database account and password.
      Number of Kafka Partitions
      Set the number of Kafka partitions. Increasing the number can improve the speed of data write and consumption. A single partition can guarantee the order of messages, while multiple partitions cannot. If you have strict requirements for the order of messages during consumption, set this value to 1.
      5. On the Subscription Type and Object page, select a subscription type and click Save.
      Subscription Type: Options include Data Update, Structure Update, and Full Instance.
      Data Update: Data updates of the selected objects are subscribed to, including INSERT, UPDATE, and DELETE operations.
      Structure Update: creation, modification, and deletion of the structures of all objects in the instance are subscribed to.
      Full Instance: Data and structure updates of all objects in the instance are subscribed to.
      Kafka Partitioning Policy: Select By table name or By table name + primary key.
      6. On the Pre-verification page, a pre-verification task will run for 2–3 minutes. After the pre-verification is passed, click Start to complete the data subscription task configuration.
      Note
      If the verification fails, fix the problem as instructed in Check Item Overview and initiate the verification again.
      7. The subscription task will be initialized, which will take 3–4 minutes. After successful initialization, the task will enter the Running status.
      8. Add a consumer group. Data subscription (Kafka Edition) allows you to create multiple consumer groups for multi-point consumption. For more information, see Creating Consumer Group. The consumption in data subscription (Kafka Edition) depends on the consumer groups of Kafka; therefore, you must create a consumer group first before data can be consumed.
      9. After the subscription instance enters the Running status, you can start consuming data. To consume data in Kafka, you need to verify the password. For specific examples, see Consuming MySQL Data. We provide demo code in multiple programming languages and descriptions of main consumption processes and key data structures.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy