tencent cloud

All product documents
Cloud Log Service
DocumentationCloud Log ServiceOperation GuideShipping and ConsumptionShipping to COSRaw Log Shipping
Raw Log Shipping
Download PDF
Last updated: 2024-01-20 17:44:35
Raw Log Shipping
Last updated: 2024-01-20 17:44:35
Download PDF

Overview

This document describes how to ship the collected raw log to COS. Currently, only logs collected in a single line, in multiple lines, or with certain separators are supported in this scenario.

Notes

You can configure CSV shipping to COS to ship certain raw logs. The following table describes the options of CSV shipping to COS but doesn't apply to JSON or Parquet shipping.
Log Collection Format
Support for Raw Log Shipping
User Raw Log
CLS Storage Format
Format for Shipping to COS
Full text in a single line
Yes. For more information, see Logs collected in a single line.
yourlog
__CONTENT__:yourlog
yourlog
Full text in multiple lines
Yes. For more information, see Logs collected in multiple lines.
yourlog
__CONTENT__:yourlog
yourlog
Separator (CSV) format
It depends. For more information, see CSV format. Only space, tab, comma, semicolon, and vertical bar are supported as raw log separators.
V1 separator V2 separator V3 separator...Vn
For example, V1,V2,V3...Vn
K1:V1 K2:V2 K3:V3...Kn:Vn
V1 separator V2 separator V3 separator...Vn
For example, V1,V2,V3...Vn
JSON format
No
K1:V1 K2:V2 K3:V3...Kn:Vn
K1:V1 K2:V2 K3:V3...Kn:Vn
V1,V2,V3...Vn differs from the original JSON.
Full regex
No
V11V22V33...Vnn
K1:V1 K2:V2 K3:V3...Kn:Vn
V1,V2,V3...Vn differs from the raw log.

Directions

Log collected in a single line or multiple lines

For logs collected in a single line or multiple lines, you can configure parameters based on CSV shipping to implement raw log shipping.
1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
2. Set Shipping Format to CSV, retain the __CONTENT__ field, delete other fields, set Separator to Space, set Escape Character to Space, set Invalid Field Filling to None, and disable Key in First Line. The parameters are described as follows:
Configuration Item
Description
Remarks
Key
__CONTENT__
For full text in a single line or multiple lines, __CONTENT__ is used as the default key, and the raw log is used as the value. When the raw log is shipped, only the __CONTENT__ field is retained.
Separator
Space
Set **Separator** to **Space** for the full text in a single line or multiple lines.
Escape Character
None
To prevent the raw log from being modified due to escape characters, set Escape Character to None.
Invalid Field Filling
None
Set Invalid Field Filling to None.
Key in First Line
Disabled
You don't need to add a description of the field name in the first line of the CSV file for raw log shipping.
3. Click OK to enable shipping.

Logs collected in CSV format

Note:
CSV shipping supports limited types of separators, including space, tab, comma, semicolon, and vertical bar. Therefore, you can ship log data in the raw format only when the separator in the raw log content is supported by CSV shipping.
1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
2. Set Shipping Format to CSV. During field configuration, you need to delete CLS metadata fields.
The parameters are described as follows:
Configuration Item
Value
Description
Key
Keys
Only the user field is retained.
Separator
A value selected from the drop-down list
Select the separator of the raw log content. If separators are different, raw log shipping is not supported. Currently, only space, tab, comma, semicolon, and vertical bar are supported.
Escape Character
None
To prevent the raw log from being modified due to escape characters, set Escape Character to None.
Invalid Field Filling
None
Set Invalid Field Filling to None.
Key in First Line
Disabled
You don't need to add a description of the field name in the first line of the CSV file for raw log shipping.
3. Click OK to enable shipping.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 available.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon