Overview
Content Delivery Network (CDN) is a very important internet infrastructure. It allows users to access various images, videos and other resources on the network quickly. During the access process, CDN will generate a large amount of log data. Through the analysis of CDN access logs, users can mine a large amount of useful information for CDN quality and performance analysis, error diagnosis, client distribution analysis, and user behavior analysis. Prerequisites
CDN logs have been collected to the Cloud Log Service (CLS). For more information, please see operation details. Scenarios
Traditional CDN log analysis
At present, CDN service providers usually provide basic monitoring metrics in real time, such as the number of requests, bandwidth, and other information. However, in many specific analysis scenarios, these default real-time metrics may not be sufficient for custom analysis requirements. Usually, users will download raw CDN logs for offline in-depth analysis and mining.
In that case, users need to set up offline analysis clusters themselves, not only requiring a lot of OPS and development costs and manpower but also making it difficult to guarantee the timeliness of offline logs in some analysis scenarios such as CDN log-based alarm reporting and troubleshooting.
CDN-to-CLS solution
Interconnect Tencent Cloud CDN and CLS so that users can ship CDN data to CLS in real time to further use the search and SQL analysis capabilities of CLS to meet users' personalized real-time log analysis needs in different scenarios:
Push-button log shipping
Analyzing tens of billions of log data entries within seconds
Visualizing real-time logs on dashboards
Real-time alarm reporting in 1 minute
Introduction to CDN logs
CDN log fields are described as follows.
|
| | | Tencent Cloud account APPID |
| | | |
| | | |
| | | Cache hit/miss. Both hits on CDN edge servers and parent nodes are marked as hit |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | Referer information, i.e., HTTP source address |
| | | Range parameter, i.e., request range |
| | | Response time (in milliseconds), which refers to the time it takes for a node to return all packets to the client after receiving a request. |
| | | A port connecting the client and CDN nodes. This field will be displayed as - if the port does not exist. |
| | | |
| | | Request timestamp in UNIX format (in seconds) |
| | | |
| | | |
| | | |
| | | CDN real-time log version |
CDN quality monitoring
Scenario 1: monitor CDN access latency and report an alarm when a specified threshold is exceeded
It is more accurate to use percentiles (such as 99% highest latency) in mathematical statistics as alarm trigger conditions. If average or individual values are used as alarm trigger conditions, the latency of some individual requests will be averaged, failing to reflect the actual situation. For example, you can use the following query analysis statement to calculate the average latency of each minute, the latency of the 50th percentile, and the latency of the 90th percentile in a one-day window (1,440 minutes).
* | select avg(request_time) as l, approx_percentile(request_time, 0.5) as p50, approx_percentile(request_time, 0.9) as p90, time_series(__TIMESTAMP__, '5m', '%Y-%m-%d %H:%i:%s', '0') as time group by time order by time desc limit 1440
If the latency of the 99th percentile is greater than 100 ms, an alarm is reported and the affected domain name, URL, and client IP are included in the alarm information to facilitate fault locating. The alarm setting statement is as follows:
* | select approx_percentile(request_time, 0.99) as p99
By configuring multidimensional analysis, you can include the affected domain name, URL, and client IP in the alarm information to help developers in fault locating.
Once an alarm is triggered, users can obtain key information via WeChat, WeCom, and SMS.
Scenario 2: monitor resource access errors and report an alarm when the period-on-period increase exceeds a specified threshold
When the number of page access errors surges, it is possible that the CDN backend server fails or is overloaded with requests. You can set an alarm to monitor the increase in the number of request errors in a certain period of time (e.g., 1 minute). When the period-on-period increase exceeds a certain threshold, an alarm is reported.
Number of errors in the recent minute
* | select * from (select * from (select * from (select date_trunc('minute', __TIMESTAMP__) as time,count(*) as errct where http_code>=400 group by time order by time desc limit 2)) order by time desc limit 1)
Number of errors in the last minute
* | select * from (select * from (select * from (select date_trunc('minute', __TIMESTAMP__) as time,count(*) as errct where http_code>=400 group by time order by time desc limit 2)) order by time asc limit 1)
The trigger condition in the alarm policy is configured as follows: Number of errors in the recent minute – Number of errors in the last minute > Specified threshold.
CDN logs contain rich content, enabling users to conduct comprehensive statistics and analysis of the overall quality and performance of CDN from multiple dimensions.
Health
Cache hit rate
Average download speed
ISPs' download count, download traffic, and download speed
Response latency
Health
Calculate the percentage of requests whose http_code is less than 500 of all requests.
* | select round(sum(case when http_code<500 then 1.00 else 0.00 end) / cast(count(*) as double) * 100,1) as "Health"
Cache hit rate
Calculate the percentage of requests whose return_code is less than 400 of requests whose value of hit is hit.
http_code<400 | select round(sum(case when hit='hit' then 1.00 else 0.00 end) / cast(count(*) as double) * 100,1) as "Cache hit rate"
Average download speed
Divide the total downloads over a period of time by the total elapsed time to get the average download speed.
* | select sum(rsp_size/1024.0) / sum(request_time/1000.0) as "Average download speed (KB/s)"
ISPs' download count, download traffic, and download speed
Use the ip_to_provider function to convert client_ip to the corresponding ISP.
* | select ip_to_provider(client_ip) as isp , sum(rsp_size)* 1.0 /(sum(request_time)+1) as "Download speed (KB/s)" , sum(rsp_size/1024.0/1024.0) as "Total download amount (MB)", count(*) as c group by isp order by c desc limit 10
Response latency
Collect access latency statistics by window, and appropriate latency time windows can be divided according to the actual situation of the application.
* | select case when request_time < 5000 then '~5s' when request_time < 6000 then '5s~6s' when request_time < 7000 then '6s~7s' when request_time < 8000 then '7~8s' when request_time < 10000 then '8~10s' when request_time < 15000 then '10~15s' else '15s~' end as latency , count(*) as count group by latency
Access errors have always been an important factor that affects service experience. When an access error occurs, you need to quickly locate the following information of the error: QPS and the proportion, domain name and URI that are affected the most, whether the error is region or ISP related, and whether the error is caused by the newly published version.
Solutions
Check the distribution of 4xx and 5xx errors.
* | select http_code , count(*) as c where http_code >= 400 group by http_code order by c desc
As shown in the figure blow, a 404 error occurs, indicating that the accessed file or content does not exist. In this case, it is necessary to check whether the resource has been deleted or terminated.
For requests whose http_code is greater than 400, we conduct multidimensional analysis, for example, sorting requests by domain name and URL separately in descending order, checking error counts by province and ISP, and checking client distribution.
Sort requests by domain name
* | select host , count(*) as count where http_code > 400 group by host order by count desc limit 10
Sort requests by URL
* | select url , count(*) as count where http_code > 400 group by url order by count desc limit 10
Check error counts by province and ISP
* | select client_ip, ip_to_province(client_ip) as "province", ip_to_provider(client_ip) as "ISP" , count(*) as "Error count" where http_code >= 400 group by client_ip order by "Error count" DESC limit 100
Check client distribution
* | select ua as "Client version", count(*) as "Error count" where http_code > 400 group by ua order by "Error count" desc limit 10
As shown in the figure, all errors occurred on the Safari client. Fault locating finds that the error is caused by a bug in the new version, which causes frequent failure to access resources via the Safari browser window.
User behavior analysis
Requirements
Where do most users come from, internally or externally?
What are the most popular resources for users?
Are there users who download massive resources frequently, and does the behavior meet expectations?
Solutions
Analyzing access sources
* | select ip_to_province(client_ip) as province , count(*) as c group by province order by c desc limit 50
Analyzing top access URLs
http_code < 400 | select url ,count(*) as "Access count", round(sum(rsp_size)/1024.0/1024.0/1024.0, 2) as "Total download amount (GB)" group by url order by "Access count" desc limit 100
Analyzing top domain names in terms of traffic (volume of data downloaded)
* | select host, sum(rsp_size/1024) as "Total Downloads" group by host order by "Total Downloads" desc limit 100
Analyzing top users in terms of download amount
* | SELECT CASE WHEN ip_to_country(client_ip)='Hong Kong' THEN concat(client_ip, ' ( Hong Kong )') WHEN ip_to_province(client_ip)='' THEN concat(client_ip, ' ( Unknown IP )') WHEN ip_to_provider(client_ip)='Intranet IP' THEN concat(client_ip, ' (Private IP )') ELSE concat(client_ip, ' ( ', ip_to_country(client_ip), '/', ip_to_province(client_ip), '/', if(ip_to_city(client_ip)='-1', 'Unknown city', ip_to_city(client_ip)), ' ',ip_to_provider(client_ip), ' )') END AS client, pv as "Total Visits", error_count as "Erroneous Access Count" , throughput as "Total Downloads(GB)" from (select client_ip , count(*) as pv, round(sum(rsp_size)/1024.0/1024/1024.0, 1) AS throughput , sum(if(http_code > 400, 1, 0)) AS error_count from log group by client_ip order by throughput desc limit 100)
Analyzing top users with valid access
* | SELECT CASE WHEN ip_to_country(client_ip)='Hong Kong' THEN concat(client_ip, ' ( Hong Kong )') WHEN ip_to_province(client_ip)='' THEN concat(client_ip, ' ( Unknown IP )') WHEN ip_to_provider(client_ip)='Intranet IP' THEN concat(client_ip, ' (Private IP )') ELSE concat(client_ip, ' ( ', ip_to_country(client_ip), '/', ip_to_province(client_ip), '/', if(ip_to_city(client_ip)='-1', 'Unknown city', ip_to_city(client_ip)), ' ',ip_to_provider(client_ip), ' )') END AS client, pv as "Total Visits", (pv - success_count) as "Erroneous Access Count" , throughput as "Total Downloads(GB)" from (select client_ip , count(*) as pv, round(sum(rsp_size)/1024.0/1024/1024.0, 1) AS throughput , sum(if(http_code < 400, 1, 0)) AS success_count from log group by client_ip order by success_count desc limit 100)
Analyzing access PVs and UVs (access count within a certain period of time and independent client IP change trend)
* | select time_series(__TIMESTAMP__, '1m', '%Y-%m-%dT%H:%i:%s+08:00', '0') as time, count(*) as pv,approx_distinct(client_ip) as uv group by time order by time limit 1000
