Scenario description
Xiaowang collects logs in JSON format to CLS (Cloud Log Service, CLS) under the following two conditions:
1. The JSON is multi-layer nested. Xiaowang wants to extract the user and App fields, where user is a secondary nested field.
2. Xiaowang's JSON log is an array and needs to split multiple logs from the array.
Raw Log
[
{
"content": {
"App": "App-1",
"start_time": "2021-10-14T02:15:08.221",
"resonsebody": {
"method": "GET",
"user": "Tom"
},
"response_code_details": "3000",
"bytes_sent": 69
}
},
{
"content": {
"App": "App-2",
"start_time": "2222-10-14T02:15:08.221",
"resonsebody": {
"method": "POST",
"user": "Jerry"
},
"response_code_details": "2222",
"bytes_sent": 1
}
}
]
{
"timestamp": 1732099684144000,
"topic": "log-containers",
"records": [
{
"category": "kube-request",
"log": "{\"requestID\":\"12345\",\"stage\":\"Complete\"}"
},
{
"category": "db-request",
"log": "{\"requestID\":\"67890\",\"stage\":\"Response\"}"
}
]
}
Processing result
[{"App":"App-1","user":"Tom"},
{"App":"App-2","user":"Jerry"}]
[
{
"category":"kube-request",
"requestID":"12345",
"stage":"Complete",
"timestamp":"1732099684144000",
"topic":"log-containers"
},
{
"category":"db-request",
"requestID":"67890",
"stage":"Response",
"timestamp":"1732099684144000",
"topic":"log-containers"
}
]
DSL Processing Function
Log 1:Processing statement
Log 2:Processing statement
//Use the ext_json function to extract structured data from JSON data, by default, it will flatten all fields
ext_json("content")
//Discard the content field
fields_drop("content")
//Discard unnecessary fields bytes_sent,method,response_code_details,start_time
fields_drop("bytes_sent","method","response_code_details","start_time")
//Split logs from the array, splitting into 2 logs
log_split_jsonarray_jmes("records")
//Discard the original field records
fields_drop("records")
//Expand the KV pairs of the log
ext_json("log")
//Discard the original field log
fields_drop("log")
