tencent cloud

All product documents
Cloud Object Storage
DocumentationCloud Object StorageConsole GuideApplication IntegrationData Export to CKafka
Data Export to CKafka
Download PDF
Last updated: 2024-06-24 16:14:43
Data Export to CKafka
Last updated: 2024-06-24 16:14:43
Download PDF

Overview

Data export to CKafka is a data export solution provided by COS based on SCF. It can help you export data in CSV, JSON, and other formats to the CKafka service in the same region for aggregation and analysis of massive amount of message and log data.

Notes

Data export to CKafka involves COS' data extraction APIs. For more information on the restrictions, see SELECT Overview.
If you have added a rule of data export to CKafka to your bucket in the COS console, the export function will appear in the SCF console. Do not delete or modify the function; otherwise, your rule may not take effect.
Currently, the data export to CKafka feature is only supported in Guangzhou, Shanghai, Beijing, and Chengdu regions.
The data export to CKafka feature depends on the SCF service, which provides a free tier. Excessive usage will be billed at SCF prices. For more information, see Pricing.

Directions

1. Log in to the COS console.
2. On the left sidebar, click App Integration > Data Export and find Data Export to CKafka.
3. Click Configure Rule to enter the rule configuration page.
4. Click Add Function.
Note:
If you haven't activated SCF, go to the SCF console to activate it and authorize the service as instructed.
5. In the pop-up window, configure the following information:

Function name prefix: It uniquely identifies a function and cannot be modified after being set. You can view the function in the SCF console.
Scenario: Select the log source from which you want to export data. We recommend you select COS log file export.
Source bucket: It is the name of the bucket where the logs are stored. If you select COS log file export, you need to enable the COS log storage feature first.
Log storage status: Make sure that the status is Enabled.
Destination Bucket: The bucket to store the logs.
Log delivery prefix: Enter a path prefix that makes it easy for you to find logs.
SCF authorization: This authorization is required, because data export to CKafka requires you to authorize SCF to read logs from your bucket.
6. Click Enter preset parameter to set the data export to CKafka configuration items as follows:

Access method: Select the CKafka access method. If you select VPC, you need to enter the VPC information.
Address: Enter your CKafka access address.
Topic: Enter the name of your CKafka topic.
Authentication configuration: Select your CKafka authentication method. If you select Authentication required, you need to enter the corresponding username and password.
Max rate: Set the rate cap for exporting data to CKafka.
7. If you want to personalize log data extraction, click Previous to configure options. Generally, we recommend you directly click Confirm to add the function.

You can perform the following operations on the created function:
Click Log to view the historical running status of data export to CKafka. If an error is reported, you can click Log to quickly redirect to the SCF console to view its details.
Click Details to view the detailed configuration of the function.
Click Edit to modify a rule of data export to CKafka.
Click Trigger and select an existing log in the bucket to directly trigger the export to CKafka.
Click Delete to delete an unwanted rule of data export to CKafka.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon