tencent cloud

피드백

Log Distribution

마지막 업데이트 시간:2024-01-20 17:44:35

    Use Case

    Tom has collected logs to CLS. The logs contain information such as the log time, log level, log content, task ID, process name, and host IP, and the information is separated by two vertical bars (||). Now Tom wants to structure the log to facilitate subsequent indexing and dashboard display. He also wants to distribute the logs to three different target log topics according to three log levels (ERROR, WARNING, and INFO) for subsequent analysis. Tom also wants the logs whose content contains the team B is working keywords to be filtered out (discarded)**.

    Scenario Analysis

    According to Tom's requirements, the processing ideas are as follows:
    1. Filter out (discard) logs that contain the team B is working keywords and place the discarded logs up front to reduce subsequent computation.
    2. Structure logs based on the separator of two vertical bars (||).
    3. Log distribution: Distribute the logs to three different target log topics according to three log levels (ERROR, WARNING, and INFO).
    Note:
    To distribute logs to multiple target log topics, you need to define the target names of the log topics when creating the data processing task. The target names will be used in the log_output("Target name") functions.

    Raw Log

    [
    {
    "message": "2021-12-09 11:34:28.279||team A is working||INFO||605c643e29e4||BIN--COMPILE||192.168.1.1"
    },
    {
    "message": "2021-12-09 11:35:28.279||team A is working ||WARNING||615c643e22e4||BIN--Java||192.168.1.1"
    },
    {
    "message": "2021-12-09 11:36:28.279||team A is working ||ERROR||635c643e22e4||BIN--Go||192.168.1.1"
    },
    {
    "message": "2021-12-09 11:37:28.279||team B is working||WARNING||665c643e22e4||BIN--Python||192.168.1.1"
    }
    ]

    DSL Processing Function

    log_drop(regex_match(v("message"),regex="team B is working",full=False))
    ext_sepstr("message","time,log,loglevel,taskId,ProcessName,ip",sep="\\|\\|")
    fields_drop("message")
    t_switch(regex_match(v("loglevel"),regex="INFO",full=True),log_output("info_log"),regex_match(v("loglevel"),regex="WARNING",full=True),log_output("warning_log"),regex_match(v("loglevel"),regex="ERROR",full=True),log_output("error_log"))

    DSL Processing Function Details

    1. Discard logs that contain the team B is working keywords. The fourth log contains the team B is working keywords and needs to be discarded.
    log_drop(regex_match(v("message"),regex="team B is working",full=False))
    2. Extract structured data based on the separator of two vertical bars (||).
    ext_sepstr("message","time,log,loglevel,taskId,ProcessName,ip",sep="\\|\\|")
    3. Discard the message field.
    fields_drop("message")
    4. According to the value of the loglevel field, INFO, WARNING, and ERROR logs will be distributed to different target log topics.
    t_switch(regex_match(v("loglevel"),regex="INFO",full=True),log_output("info_log"),regex_match(v("loglevel"),regex="WARNING",full=True),log_output("warning_log"),regex_match(v("loglevel"),regex="ERROR",full=True),log_output("error_log"))

    Processing Result

    Note:
    Target log topics and target names must be configured in advance.
    The following log is distributed to info_log (Data processing-target 3). See the mappings between target names and log topics in the figure above.
    {"ProcessName":"BIN--COMPILE","ip":"192.168.1.1","log":"team A is working","loglevel":"INFO","taskId":"605c643e29e4","time":"2021-12-09 11:34:28.279"}
    The following log is distributed to warning_log (Data processing-target 2).
    {"ProcessName":"BIN--COMPILE","ip":"192.168.1.1","log":"team A is working","loglevel":"INFO","taskId":"605c643e29e4","time":"2021-12-09 11:34:28.279"}
    The following log is distributed to error_log (Data processing-target 1).
    {"ProcessName":"BIN--Go","ip":"192.168.1.1","log":"team A is working ","loglevel":"ERROR","taskId":"635c643e22e4","time":"2021-12-09 11:36:28.279"}
    
    문의하기

    고객의 업무에 전용 서비스를 제공해드립니다.

    기술 지원

    더 많은 도움이 필요하시면, 티켓을 통해 연락 바랍니다. 티켓 서비스는 연중무휴 24시간 제공됩니다.

    연중무휴 24시간 전화 지원