tencent cloud

Log Structuring - Grok
最終更新日:2024-12-18 16:36:30
Log Structuring - Grok
最終更新日: 2024-12-18 16:36:30

Scenario description

XiaoWang reports the logs collected by Beats to CLS through Kafka protocol to upload logs. The approach is as follows:
1. Use the grok function to structure the logs.
2. Use the time field in the logs to replace the log time of CLS(__TIMESTAMP__).




Raw Log


  {
    "__FILENAME__": "",
    "__SOURCE__": "192.168.100.123",
    "message": "2024-10-11 15:32:10.003 DEBUG [gateway,746db87efd1bbcf5434cb9835c59e522,47c3036810e0c33b] [scheduled-Thread-1] c.i.g.c.f.d.a.task.AppleHealthCheckTask"
  }

Processing result

{
"__FILENAME__":"",
"__SOURCE__":"192.168.100.123",
"__TIMESTAMP__":"1728631930003",
"level":"DEBUG",
"service":"gateway",
"spanid":"47c3036810e0c33b",
"time":"2024-10-11 15:32:10.003",
"traceid":"746db87efd1bbcf5434cb9835c59e522"
}

Processing statement

// Use the grok function to extract time, log level, service, traceid, and spanid from the logs
ext_grok("message",grok="%{TIMESTAMP_ISO8601:time} %{DATA:level} \\[%{DATA:service},%{DATA:traceid},%{DATA:spanid}\\]")
// Delete message field
fields_drop("message")
// custom_cls_log_time function, use the new field time to replace the log time of CLS (__TIMESTAMP__)
custom_cls_log_time(dt_to_timestamp(v("time"), zone="UTC+8"))


この記事はお役に立ちましたか？
営業担当者に お問い合わせ いただくか チケットを提出 してサポートを求めることができます。
はい
いいえ

フィードバック