- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Metric Collection
- Log Storage
- Metric Storage
- Search and Analysis (Log Topic)
- Overview of Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Multi-Topic Search
- Context Search and Analysis
- Custom Redirect
- Downloading Log
- Search and Analysis (Metric Topic)
- Dashboard
- Data Processing documents
- Data Processing
- Data Processing Overview
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Practical Tutorial
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Metric Collection
- Log Storage
- Metric Storage
- Search and Analysis (Log Topic)
- Overview of Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Multi-Topic Search
- Context Search and Analysis
- Custom Redirect
- Downloading Log
- Search and Analysis (Metric Topic)
- Dashboard
- Data Processing documents
- Data Processing
- Data Processing Overview
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Practical Tutorial
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
Overview
Cloud Object Storage (COS) access logs record information about users' access to COS resources, including object upload (
PUT), object deletion (DELETE), and object getting (GET). By analyzing access logs, you can perform audit backtracking, such as deleting resource records and collecting statistics on popular resources. This document introduces how to analyze COS access logs.Prerequisite
COS logs have been collected to Cloud Log Service (CLS). For more information, please see Enabling Real-Time Log Feature on COS.
Introduction to Access Logs
COS access logs record information such as the source bucket, user ID, and request method.
No. | Field | Description | Example |
1 | eventVersion | Log version | 1.0 |
2 | bucketName | Bucket name | examplebucket-1250000000 |
3 | qcsRegion | Request region | ap-beijing |
4 | eventTime | Event time (request end time, which is a timestamp in UTC+0 time zone) | 2018-12-01T11:02:33Z |
5 | eventSource | Access domain name | examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com |
6 | eventName | Event name | UploadPart |
7 | remoteIp | Source IP | 192.168.0.1 |
8 | userSecretKeyId | User access KeyId | AKIDNYVCdoJQyGJ5brTf |
9 | reservedField | Reserved field | Displayed as - |
10 | reqBytesSent | Request bytes | 83886080 |
11 | deltaDataSize | Change in storage made by the request (in bytes) | 808 |
12 | reqPath | Requested file path | /folder/text.txt |
13 | reqMethod | Request method | put |
14 | userAgent | User agent (UA) | cos-go-sdk-v5.2.9 |
15 | resHttpCode | HTTP return code | 404 |
16 | resErrorCode | Error code | NoSuchKey |
17 | resErrorMsg | Error message | The specified key does not exist. |
18 | resBytesSent | Bytes returned | 197 |
19 | resTotalTime | Total time used by the request (in milliseconds, i.e., the time between the last byte of the response and the first byte of the request) | 4295 |
20 | logSourceType | Source type of the log | USER (user access requests), CDN (CDN origin-pull requests) |
21 | storageClass | Storage class | STANDARD, STANDARD_IA, ARCHIVE |
22 | accountId | Bucket owner ID | 100000000001 |
23 | resTurnAroundTime | Time used by the request server (in milliseconds, i.e., the time between the first byte of the response and the last byte of the request) | 4295 |
24 | requester | Requester | Root account ID, sub-account ID, or - (anonymous access) |
25 | requestId | Request ID | NWQ1ZjY4MTBfMjZiMjU4NjRfOWI1N180NDBiYTY= |
26 | objectSize | Object size, in bytes | 808. If you use multipart upload, objectSize will only be displayed when the upload is completed, and will be - during the multipart upload process |
27 | versionId | Object version ID | Random string |
28 | targetStorageClass | Destination storage class, recorded for replication requests | STANDARD, STANDARD_IA, ARCHIVE |
29 | referer | HTTP referer of the request | *.example.com or 111.111.111.1 |
30 | requestUri | Request URI | "GET /fdgfdgsf%20/%E6%B5%AE%E7%82%B9%E6%95%B0 HTTP/1.1" |
Examples
Example 1: audit backtracking
Requirement
An object file cannot be accessed, and the cause needs to be located.
Solution
Go to the COS access log search page, and enter the object name as the keyword to search for logs.
json-log2019-05-09_00645d9a-1118-4d69-8411-cfd57ede9ea1_000
According to the time column chart, 14 logs are recorded on the last day. For the drill-down analysis of the 14 log records, click the quick analysis bar on the left to view the resHttpCode information.
According to the quick analysis, there are 6 request log records whose resHttpCode is not 200: resHttpCode is 403 for 5 log records and 204 for 1 log record. Click to search for these logs quickly.
According to the logs, the 5 log records whose error code is Access Deny are object access failure logs. According to the logs whose resHttpCode is 204, object access failed because user
1000****** performed object deletion at around 20:16 on August 24 in the COS console.Example 2: operations statistics
Requirement
Collect statistics on the top 10 most visited buckets of the day
Collect statistics on the access trend of a certain bucket
Collect statistics on the top 10 visitors of the error requests
Collect statistics on the bucket distribution of the failed operations
User request efficiency trend
Solution
Collect statistics on the top 10 most visited buckets of the day
(reqMethod:"GET") | select bucketName, count(*) group by bucketName
Collect statistics on the access trend of a certain bucket
* | select time_series(TIMESTAMP, '1m', '%Y-%m-%dT%H:%i:%s+08:00', '0') AS time, count(*) as pv, reqMethod group by time, reqMethod order by time limit 200
Collect statistics on the top 10 visitors of the error requests
resHttpCode:>200 | select remoteIp, count(*) group by remoteIp
Collect statistics on the bucket distribution of the failed operations
resHttpCode:>200 | select bucketName, count(*) group by bucketName
User request efficiency trend
* | select time_series(__TIMESTAMP__, '5m', '%Y-%m-%d %H:%i:%s', '0') as time,round(sum(case when resHttpCode=200 then 1.00 else 0.00 end) / cast(count(*) as double) * 100,1) as "Request Efficiency" group by time limit 1000
User request source distribution
* | select ip_to_province(remoteIp) as province , count(*) as c group by province order by c desc limit 50
Yes
No
Was this page helpful?