tencent cloud

All product documents
Cloud Log Service
DocumentationCloud Log ServiceOperation GuideData Processing documentsData ProcessingProcessing CasesLog Structuring - Grok
Log Structuring - Grok
Download PDF
Last updated: 2024-12-18 16:36:30
Log Structuring - Grok
Last updated: 2024-12-18 16:36:30
Download PDF

Scenario description

XiaoWang reports the logs collected by Beats to CLS through Kafka protocol to upload logs. The approach is as follows:
1. Use the grok function to structure the logs.
2. Use the time field in the logs to replace the log time of CLS(__TIMESTAMP__).




Raw Log


  {
    "__FILENAME__": "",
    "__SOURCE__": "192.168.100.123",
    "message": "2024-10-11 15:32:10.003 DEBUG [gateway,746db87efd1bbcf5434cb9835c59e522,47c3036810e0c33b] [scheduled-Thread-1] c.i.g.c.f.d.a.task.AppleHealthCheckTask"
  }

Processing result

{
"__FILENAME__":"",
"__SOURCE__":"192.168.100.123",
"__TIMESTAMP__":"1728631930003",
"level":"DEBUG",
"service":"gateway",
"spanid":"47c3036810e0c33b",
"time":"2024-10-11 15:32:10.003",
"traceid":"746db87efd1bbcf5434cb9835c59e522"
}

Processing statement

// Use the grok function to extract time, log level, service, traceid, and spanid from the logs
ext_grok("message",grok="%{TIMESTAMP_ISO8601:time} %{DATA:level} \\[%{DATA:service},%{DATA:traceid},%{DATA:spanid}\\]")
// Delete message field
fields_drop("message")
// custom_cls_log_time function, use the new field time to replace the log time of CLS (__TIMESTAMP__)
custom_cls_log_time(dt_to_timestamp(v("time"), zone="UTC+8"))


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon