tencent cloud

All product documents
TDMQ for CKafka
DocumentationTDMQ for CKafkaCKafka ConnectorTask ManagementCreating Data Distribution TaskData TargetCOS
COS
Download PDF
Last updated: 2024-09-09 21:57:25
COS
Last updated: 2024-09-09 21:57:25
Download PDF

Overview

The CKafka Connector provides data distribution capabilities, allowing you to distribute CKafka data to COS for analysis, download and other operations.

Prerequisites

Currently, this feature depends on the COS service, and you should enable the relevant product features to use it.
Ensure that the target bucket for data distribution has been created.

Directions

Creating a Task

1. Log in to the CKafka Console .
2. In the left sidebar, click Connectors > Task List , select the right region, and then click Create Task .
3. Fill in the task name, select Data Distribution as the task type, select Cloud Object Storage (COS)as the data target type,then click Next .++
4. Configure the data source information.

Topic Type: Select the data source Topic.
Elastic Topic: Select the pre-created elastic Topic, for details see Topic Management.
CKafka Instance Topic: Select the instance and Topic created in CKafka. If the instance has ACL policies configured, ensure the selected topic has read and write permissions. For details see Creating Topic.
Start Offset: Select how to handle historical messages during the dump by setting the Topic offset.
5. After setting the above information, click Next , click Preview Topic Message , and the first message from the source topic will be obtained and parsed.
Note
Currently, message parsing should meet the following requirements:
The message is a JSON string.
The source data should be in a single-layer JSON format. For nested JSON, you can useData Processing to perform simple message format conversion.
6. (Optional) Toggle on the Process Source Data for source data. For detailed configuration, see Simple Data Processing.
7. Click Next to configure the data target information.

Target Bucket: Select the target bucket for data delivery.
Directory Prefix: Supports custom directory prefixes. Log files will be delivered to this directory in the COS Bucket.
Partition Format: Automatically generates directories based on the task creation time using the syntax of strftime, where the slash / indicates a first-level COS directory. Fill out the partition format according to the strftime format requirements. Different partition formats will affect the file path delivered to COS. Below is an example of how to use the partition format, e.g., if the data is delivered to the bucket_test bucket, the directory prefix is logset/ ; for a delivery time of 2018/7/31 17:14, the corresponding delivery file path is as follows:
Bucket Name
Directory Prefix
Partition Format
COS File Path
bucket_test
logset/
%Y/%m/%d
bucket_test:logset/2018/7/31_{random}_{index}
bucket_test
logset/
%Y%m%d/%H
bucket_test:logset/20180731/14_{random}_{index}
bucket_test
logset/
%Y%m%d/log
bucket_test:logset/20180731/log_{random}_{index}
Aggregation Method:
Single File Size: Specify the maximum size of an uncompressed delivery file within the delivery interval. If the log file exceeds this limit within the interval, it will be split into multiple log files.
By Time: Select an aggregation interval based on message volume, ranging from 1 minute to 24 hours.
Storage Format: Supports CSV and JSON formats.
Role Authorization: To use COS product features, you need to grant a third-party role the permission to access to related product on your behalf.
8. Click Submit . You can see the created task in the task list and check its creation progress in the status bar.


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon