In simple Ops scenarios, logs are directly output to local files on the server, and then you can run the grep
command in Linux to search for logs. In complex service systems, logs are scattered on different servers, CLI operations are not intuitive, and server permission management is restricted. As a result, it is difficult to find logs, which seriously affects Ops efficiency. It's even more difficult if you need to do some statistical analysis or monitor alarms based on logs.
This document describes how to quickly migrate the local logs searched by the grep
command to CLS to obtain the following advantages:
Data is stored and searched centrally, and you don't need to log in to multiple servers to query it individually, which is critical in load balancing, microservice, and other architectures.
You can quickly search for logs with a few clicks, eliminating command lines and tedious server permission management.
You can perform statistical analysis based on logs to obtain key business metrics such as PV, API response time, and API error rate.
You can detect exceptional logs in real time and receive notifications through various methods such as SMS, email, and WeChat.
Note:
If your logs have been collected to CLS, you can skip the steps of log collection and index configuration and go to Step 3. Searching for logs. Step 1. Collect Logs
For server local logs, you can use LogListener to collect raw logs to CLS. For how to install LogListener, please see LogListener Installation Guide.
If you are using Tencent Cloud CVMs, you can also use the automatic installation feature provided by the console to install LogListener. For more information, please see Deploying LogListener on CVMs in Batches. Different from storing logs locally on servers, collecting logs to CLS makes log search and statistical analysis easier. When collecting logs, you can convert non-formatted raw logs to formatted data. Assume that the raw log is as follows:
10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/
You can use the separator :::
to split the log into eight fields and name each field as follows:
IP: 10.20.20.10
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
For operation details, please see Separator Format. In addition to using separators to split logs, you can use regular expression, JSON, and full-text log splitting methods. For more information, please see Collecting Text Logs. The purpose of configuring indexes is to define searchable fields and their types to facilitate future log search. For most scenarios, you can use the automatic index configuration feature to complete the configuration with a few clicks. For more information, please see Configuring Indexes. Step 3. Search for Logs
The following uses the commonly used grep
command as an example to describe how to achieve a similar log search feature in CLS.
Assume the raw log is as follows:
10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/
The corresponding formatted log in CLS is as follows:
IP: 10.20.20.10
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
Example 1
Search for logs where request
is /online/sample
.
Use the grep
command
Use the CLS search mode
Example 2
Search for logs where status
(status code) is not 200
.
Use the grep
command
In fact, this mode may exclude certain logs where the number 200 exists but status
is not 200
.
Use the CLS search mode
CLS also supports other more flexible search methods. For example, you can search for logs where status
is greater than or equal to 500 as follows:
Example 3
Count the number of logs where status
is not 200
.
Use the grep
command
Use the CLS search mode
NOT status:200 | select count(*) as errorLogCounts
Example 4
Search for logs where status
is 200
and request
is /online/sample
.
Use the grep
command
Use the CLS search mode
status:200 AND request:"/online/sample"
Example 5
Search for logs where request
is /online/sample
or /offline/sample
.
Use the grep
command
Use the CLS search mode
request:"/online/sample" OR request:"/offline/sample"
Example 6
Search for logs where request
is /online/sample
but the log file is not test.log
.
Use the grep
command
Use the CLS search mode
request:"/online/sample" AND NOT __FILENAME__:"test.log"
Example 7
Search for the first 10 lines in logs where time
is [Tue Jan 22 14:49:45 CST 2019 +0800]
.
Use the grep
command
Use the CLS search mode
time:"[Tue Jan 22 14:49:45 CST 2019 +0800]"
If a matching log is found, you can click Context Search in the console to view logs near the log found.
Was this page helpful?