tencent cloud

All product documents
TDMQ for CKafka
DocumentationTDMQ for CKafkaSDK DocumentationSDK for ConnectorElastic Topic Message Sending and ReceivingConnecting Filebeats to CKafka
Connecting Filebeats to CKafka
Download PDF
Last updated: 2024-09-09 21:29:32
Connecting Filebeats to CKafka
Last updated: 2024-09-09 21:29:32
Download PDF
Beats platform integrates a variety of single-purpose data collectors. Once installed, these collectors can serve as lightweight agents, sending collected data from hundreds or thousands of machines to a specified target.

Beats offers multiple collectors, and you can download the appropriate collector according to your needs. This document will introduce how to connect Filebeat (a lightweight log collector) to CKafka and solutions to FAQs encountered after connection.

Prerequisites

Download and install Filebeat (see Download Filebeat)
Download and install JDK 8 (see Download JDK 8)

Directions

Step 1: Preparations

1. On the Elastic Topic list page of the console, create a Topic.

2. Click the ID of the Topic to enter the Basic Information page and obtain the username, password, and address information.

3. In the Subscription Relationships tab, create a subscription relationship (consumption group).


Step 2: Preparing the Configuration File

Enter the Filebeat installation directory and create the configuration monitoring file named filebeat.yml.
#======= For versions later than Filebeat 7.x, change `filebeat.prospectors` to `filebeat.inputs` =======
filebeat.prospectors:


- input_type: log 

# This is the monitoring file path.
  paths:
    - /var/log/messages

#=======  Outputs =========

#------------------ kafka -------------------------------------
output.kafka:
  version:0.10.2 // Configure them according to different CKafka instance open-source versions.
  # Set the CKafka connection address.
  hosts: ["xx.xx.xx.xx:xxxx"]
  # Set the name of the target topic.
  topic: 'test'
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: none
  max_message_bytes: 1000000

  # SASL requires the following information to be configured. If SASL is not needed, you can skip the following two options.
  username: "yourusername"  
  password: "yourpassword"
Parameter
Description
host
The connection address. It can be obtained from the basic information page of an elastic Topic in the console.

username
The username. It can be obtained from the basic information page of an elastic Topic in the console.
password
The user password. It can be obtained from the basic information page of an elastic Topic in the console.
topic
The topic name. It can be obtained from the basic information page of an elastic Topic in the console.

Step 3: Sending Messages with Filebeat

1. Run the following command to start the client.
sudo ./filebeat -e -c filebeat.yml
2. Add data to the monitoring file (example: write to the monitored testlog file).
    echo ckafka1 >> testlog
    echo ckafka2 >> testlog
    echo ckafka3 >> testlog
3. Start the Consumer to consume the corresponding Topic and get the following data.
    {"@timestamp":"2017-09-29T10:01:27.936Z","beat":{"hostname":"10.193.9.26","name":"10.193.9.26","version":"5.6.2"},"input_type":"log","message":"ckafka1","offset":500,"source":"/data/ryanyyang/hcmq/beats/filebeat-5.6.2-linux-x86_64/testlog","type":"log"}
    {"@timestamp":"2017-09-29T10:01:30.936Z","beat":{"hostname":"10.193.9.26","name":"10.193.9.26","version":"5.6.2"},"input_type":"log","message":"ckafka2","offset":508,"source":"/data/ryanyyang/hcmq/beats/filebeat-5.6.2-linux-x86_64/testlog","type":"log"}
    {"@timestamp":"2017-09-29T10:01:33.937Z","beat":{"hostname":"10.193.9.26","name":"10.193.9.26","version":"5.6.2"},"input_type":"log","message":"ckafka3","offset":516,"source":"/data/ryanyyang/hcmq/beats/filebeat-5.6.2-linux-x86_64/testlog","type":"log"}

SASL/PLAINTEXT Mode

If you want to configure SASL/PLAINTEXT, you need to set the username and password under the Kafka configuration.
 # SASL requires the following information to be configured. If SASL is not needed, you can skip the next two options.
  username: "yourusername"  
  password: "yourpassword"

FAQs

If you find a large number of INFO logs in the Filebeat logs (default path /var/log/filebeat/filebeat), such as:
2019-03-20T08:55:02.198+0800    INFO    kafka/log.go:53 producer/broker/544 starting up
2019-03-20T08:55:02.198+0800    INFO    kafka/log.go:53 producer/broker/544 state change to [open] on wp-news-filebeat/4
2019-03-20T08:55:02.198+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/4 selected broker 544
2019-03-20T08:55:02.198+0800    INFO    kafka/log.go:53 producer/broker/478 state change to [closing] because EOF
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 Closed connection to broker bitar1d12:9092
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/5 state change to [retrying-3]
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/4 state change to [flushing-3]
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/5 abandoning broker 478
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/2 state change to [retrying-2]
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/2 abandoning broker 541
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/leader/wp-news-filebeat/3 state change to [retrying-2]
2019-03-20T08:55:02.199+0800    INFO    kafka/log.go:53 producer/broker/478 shut down
A large number of INFO logs may indicate a potential issue with the Filebeat version. Products in the Elastic family are updated frequently, and different major versions often have compatibility issues. For example, v6.5.x supports Kafka v0.9, v0.10, v1.1.0, and v2.0.0 by default, while v5.6.x supports Kafka v0.8.2.0 by default.
You should check the version configuration in the configuration file:
output.kafka:
  version:0.10.2 // Configure them according to different CKafka instance open-source versions.


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon