tencent cloud

Feedback

Collection Overview

Last updated: 2024-01-20 17:14:28

    Overview

    CLS provides log collection clients that allow you to collect your application logs and import them into CLS through APIs or SDKs. Currently, CLS requires logs to be uploaded in a structured manner as key-value pairs.

    Log Structuring

    The structuring of logs is to store your log data on the CLS platform in key-value format. Structured logs can be searched for, analyzed, and shipped based on specified keys. CLS allows you to report structured data directly. For more information, see the following:
    For example, a local raw log is as follows:
    10.20.20.10;[Tue Jan 22 14:49:45 CST 2019 +0800];GET /online/sample HTTP/1.1;127.0.0.1;200;647;35;http://127.0.0.1/
    Specify that the log is parsed by separator and select semicolon (;) as the separator. Then the log can be parsed into multiple field groups and each group is organized in key-value pairs. A key name is defined for each key as follows:
    IP: 10.20.20.10
    time: [Tue Jan 22 14:49:45 CST 2019 +0800]
    request: GET /online/sample HTTP/1.1
    host: 127.0.0.1
    status: 200
    length: 647
    bytes: 35
    referer: http://127.0.0.1/
    LogListener provides various parsing modes, in which reported logs with full text in a single line and full text in multi lines are both structured. LogListener adds __CONTENT__ as a key and uses the original text as the value by default. The key-value pair is as follows:
    LogListener Parsing Mode
    Key
    Description
    Full text in a single line/Full text in multi lines
    __CONTENT__
    __CONTENT__ is the key name by default.
    JSON format
    Name in the JSON file
    The name and value in the original text of the JSON log are used as a key-value pair.
    Separator format
    Custom
    After dividing fields using separators, you need to define a key name for each field group.
    Full regular expression
    Custom
    After extracting fields based on a regular expression, you need to define a key name for each field group.

    Collection Methods

    CLS provides multiple methods for data collection:
    Collection Method
    Description
    API
    You can call CLS APIs to upload structured logs to CLS. For more information, see Uploading Log via API.
    SDK
    You can use SDKs to upload structured logs to CLS. For more information, see Collection via SDK.
    LogListener client
    LogListener is a log collection client provided by CLS. You can quickly access CLS by simply configuring LogListener in the console. For more information, see LogListener Use Process.
    A comparison of the collection methods is as follows:
    Category
    Collection via LogListener
    Collection via API
    Code modification
    Provides a non-intrusive collection method for applications, without code modification.
    Reports logs only after modifying application code.
    Resumable upload
    Supports resumable upload of logs.
    Automatically implemented by code.
    Retransmission upon failure
    Provides an inherent retry mechanism.
    Automatically implemented by code.
    Local cache
    Supports local cache, ensuring data integrity during peak hours.
    Automatically implemented by code.
    Resource occupation
    Occupies resources such as memory and CPU resources.
    Occupies no additional resources.

    Accessing Log Sources

    You can access different log sources in different ways. For more information, see the following tables:
    Log source environment
    System Environment
    Recommended Access Method
    Linux/Unix
    LogListener collection / Upload over Kafka / Log upload via API
    Windows
    iOS/Android/Web
    Tencent Cloud service logs
    For more information, see Tencent Cloud Service Log Access.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support