tencent cloud

All product documents
Cloud Log Service
DocumentationCloud Log ServiceOperation GuideData Processing documentsData ProcessingProcessing CasesSingle-Line Text Log Structuration
Single-Line Text Log Structuration
Download PDF
Last updated: 2024-01-20 17:44:35
Single-Line Text Log Structuration
Last updated: 2024-01-20 17:44:35
Download PDF

Use Case

Tom has collected a log to CLS. The log does not use a fixed separator and is in single-line text format. Now Tom wants to structuralize the log and extract the log time, log level, operation, and URL information from the text for subsequent search and analysis.

Scenario Analysis

According to Tom's requirements, the processing ideas are as follows:1.The content in {...} is the detailed information of operations and can be extracted with a regular expression.2. Use a regular expression to extract the log time, log level, and URL.

Raw Log

{
    "content": "[2021-11-24 11:11:08,232][328495eb-b562-478f-9d5d-3bf7e][INFO] curl -H 'Host: ' http://abc.com:8080/pc/api -d {\"version\": \"1.0\",\"user\": \"CGW\",\"password\": \"123\",\"interface\": {\"Name\": \"ListDetail\",\"para\": {\"owner\": \"1253\",\"orderField\": \"createTime\"}}}"
}

DSL Processing Function

fields_set("Action",regex_select(v("content"),regex="\{[^\}]+\}",index=0,group=0)) 
fields_set("loglevel",regex_select(v("content"),regex="\[[A-Z]{4}\]",index=0,group=0)). 
fields_set("logtime",regex_select(v("content"),regex="\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}",index=0,group=0))
fields_set("Url",regex_select(v("content"),regex="([a-z]{3}).([a-z]{3}):([0-9]{4})",index=0,group=0))
fields_drop("content")

DSL Processing Function Details

1. Create a field named Action and use the regular expression {[^}]+} to match {...}. 
fields_set("Action",regex_select(v("content"),regex="\{[^\}]+\}",index=0,group=0))
2. Create a field named loglevel and use the regular expression [A-Z]{4} to match INFO.
fields_set("loglevel",regex_select(v("content"),regex="\[[A-Z]{4}\]",index=0,group=0)).
3. Create a field named logtime and use the regular expression d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} to match 2021-11-24 11:11:08.
fields_set("logtime",regex_select(v("content"),regex="\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}",index=0,group=0))
4. Create a field named Url, use the regular expression [a-z]{3}.[a-z]{3} to match abc.com, and use [0-9]{4} to match 8080. 
fields_set("Url",regex_select(v("content"),regex="([a-z]{3}).([a-z]{3}):([0-9]{4})",index=0,group=0))
5. Discard the content field. 
fields_drop("content")

Processing Result

{"Action":"{\"version\": \"1.0\",\"user\": \"CGW\",\"password\": \"123\",\"interface\": {\"Name\": \"ListDetail\",\"para\": {\"owner\": \"1253\",\"orderField\": \"createTime\"}","Url":"abc.com:8080","loglevel":"[INFO]","logtime":"2021-11-24 11:11:08,232"}

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon