The system architecture of the CKafka Connector is as shown below:
Primarily divided into four layers:
Data Source
The data source refers to the location of the customer's data source. The data source can be in the cloud, self-built IDC, cross cloud, or hybrid cloud. The data can be business data, log data, or data in the DB, etc.
Access Layer
The CKafka connector provides an access layer that adapts to various protocols, such as HTTP Rest, Kafka Protocol, Change Data Capture, etc. The access layer is deployed in a distributed manner, which features capabilities such as Auto Scaling and automatic retrying, ensuring the stability of data access.
Storage Layer
The storage layer of the CKafka connector is the Message Queue (MQ) on Tencent Cloud. By default, it supports Kafka, but it can also support other MQs such as Pulsar, RocketMQ. The MQ storage layer primarily serves to level peak loads, distribute data, and cache.
Data Distribution Layer
The CKafka Connector offers a data cleansing (ETL) engine that, based on the configuration, carries out data cleansing and provides the Connector with the dump capability. It can consume data from the data storage layer and guide the data into various different downstream storage engines.
