The CKafka Connector provides data distribution capabilities, allowing you to distribute CKafka data to COS for analysis, download and other operations.
Prerequisites
Currently, this feature depends on the COS service, and you should enable the relevant product features to use it.
Ensure that the target bucket for data distribution has been created.
2. In the left sidebar, click Connectors > Task List , select the right region, and then click Create Task .
3. Fill in the task name, select Data Distribution as the task type, select Cloud Object Storage (COS)as the data target type,then click Next .++
4. Configure the data source information.
Topic Type: Select the data source Topic.
Elastic Topic: Select the pre-created elastic Topic, for details see Topic Management.
CKafka Instance Topic: Select the instance and Topic created in CKafka. If the instance has ACL policies configured, ensure the selected topic has read and write permissions. For details see Creating Topic.
Start Offset: Select how to handle historical messages during the dump by setting the Topic offset.
5. After setting the above information, click Next , click Preview Topic Message , and the first message from the source topic will be obtained and parsed.
Note
Currently, message parsing should meet the following requirements:
The message is a JSON string.
The source data should be in a single-layer JSON format. For nested JSON, you can useData Processing to perform simple message format conversion.
6. (Optional) Toggle on the Process Source Data for source data. For detailed configuration, see Simple Data Processing.
7. Click Next to configure the data target information.
Target Bucket: Select the target bucket for data delivery.
Directory Prefix: Supports custom directory prefixes. Log files will be delivered to this directory in the COS Bucket.
Partition Format: Automatically generates directories based on the task creation time using the syntax of strftime, where the slash / indicates a first-level COS directory.
Fill out the partition format according to the strftime format requirements. Different partition formats will affect the file path delivered to COS. Below is an example of how to use the partition format, e.g., if the data is delivered to the bucket_test bucket, the directory prefix is logset/ ; for a delivery time of 2018/7/31 17:14, the corresponding delivery file path is as follows:
Bucket Name
Directory Prefix
Partition Format
COS File Path
bucket_test
logset/
%Y/%m/%d
bucket_test:logset/2018/7/31_{random}_{index}
bucket_test
logset/
%Y%m%d/%H
bucket_test:logset/20180731/14_{random}_{index}
bucket_test
logset/
%Y%m%d/log
bucket_test:logset/20180731/log_{random}_{index}
Aggregation Method:
Single File Size: Specify the maximum size of an uncompressed delivery file within the delivery interval. If the log file exceeds this limit within the interval, it will be split into multiple log files.
By Time: Select an aggregation interval based on message volume, ranging from 1 minute to 24 hours.
Storage Format: Supports CSV and JSON formats.
Role Authorization: To use COS product features, you need to grant a third-party role the permission to access to related product on your behalf.
8. Click Submit . You can see the created task in the task list and check its creation progress in the status bar.
Yes
No
Was this page helpful?