- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- EMR on CVM Operation Guide
- Planning Cluster
- Configuring Cluster
- Managing Cluster
- Instance Information
- Node Specification Management
- Checking and Updating Public IP
- Logging in to Clusters
- Cluster Scale-Out
- Cluster Scale-in
- Releasing Now
- Auto Scaling
- Graceful Scale-In
- Changing Configurations
- Automatic Replacement
- Exporting Software Configuration
- Cluster Scripts
- Cluster Termination
- Operation Logs
- Task Center
- Disk Management
- Managing Service
- Monitoring and Alarms
- Cluster Overview
- Node Status
- Service Status
- Cluster Event
- Log
- Application Analysis
- Cluster Inspection
- Java Analysis
- Monitoring Metrics
- Node Monitoring Metrics
- HDFS Monitoring Metrics
- YARN Monitoring Metrics
- ZooKeeper Monitoring Metrics
- HBase Monitoring Metrics
- Hive Monitoring Metrics
- Spark Monitoring Metrics
- Presto Monitoring Metrics
- Trino Monitoring Metrics
- ClickHouse Monitoring Metrics
- Druid Monitoring Metrics
- Kudu Monitoring Metrics
- Alluxio Monitoring Metrics
- PrestoSQL Monitoring Metrics
- Impala Monitoring Metrics
- Ranger Monitoring Metrics
- COSRanger Monitoring Metrics
- Doris Monitoring Metrics
- Kylin Monitoring Metrics
- Zeppelin Monitoring Metrics
- Oozie Monitoring Metrics
- Storm Monitoring Metrics
- Livy Monitoring Metrics
- Kyuubi Monitoring Metrics
- StarRocks Monitoring Metrics
- Kafka Monitoring Metrics
- Alarm Configurations
- Alarm Records
- EMR on TKE Operation Guide
- EMR Serverless HBase Operation Guide
- EMR Development Guide
- Hadoop Development Guide
- HDFS Common Operations
- HDFS Federation Management Development Guide
- HDFS Federation Management
- Submitting MapReduce Tasks
- Automatically Adding Task Nodes Without Assigning ApplicationMasters
- YARN Task Queue Management
- Practices on YARN Label Scheduling
- Hadoop Practical Tutorial
- Using API to Analyze Data in HDFS and COS
- Dumping YARN Job Logs to COS
- Spark Development Guide
- Hbase Development Guide
- Phoenix on Hbase Development Guide
- Hive Development Guide
- Presto Development Guide
- Sqoop Development Guide
- Hue Development Guide
- Oozie Development Guide
- Flume Development Guide
- Kerberos Development Guide
- Knox Development Guide
- Alluxio Development Guide
- Kylin Development Guide
- Livy Development Guide
- Kyuubi Development Guide
- Zeppelin Development Guide
- Hudi Development Guide
- Superset Development Guide
- Impala Development Guide
- Druid Development Guide
- TensorFlow Development Guide
- Kudu Development Guide
- Ranger Development Guide
- Kafka Development Guide
- Iceberg Development Guide
- StarRocks Development Guide
- Flink Development Guide
- Hadoop Development Guide
- Practical Tutorial
- API Documentation
- FAQs
- Service Level Agreement
- Contact Us
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- EMR on CVM Operation Guide
- Planning Cluster
- Configuring Cluster
- Managing Cluster
- Instance Information
- Node Specification Management
- Checking and Updating Public IP
- Logging in to Clusters
- Cluster Scale-Out
- Cluster Scale-in
- Releasing Now
- Auto Scaling
- Graceful Scale-In
- Changing Configurations
- Automatic Replacement
- Exporting Software Configuration
- Cluster Scripts
- Cluster Termination
- Operation Logs
- Task Center
- Disk Management
- Managing Service
- Monitoring and Alarms
- Cluster Overview
- Node Status
- Service Status
- Cluster Event
- Log
- Application Analysis
- Cluster Inspection
- Java Analysis
- Monitoring Metrics
- Node Monitoring Metrics
- HDFS Monitoring Metrics
- YARN Monitoring Metrics
- ZooKeeper Monitoring Metrics
- HBase Monitoring Metrics
- Hive Monitoring Metrics
- Spark Monitoring Metrics
- Presto Monitoring Metrics
- Trino Monitoring Metrics
- ClickHouse Monitoring Metrics
- Druid Monitoring Metrics
- Kudu Monitoring Metrics
- Alluxio Monitoring Metrics
- PrestoSQL Monitoring Metrics
- Impala Monitoring Metrics
- Ranger Monitoring Metrics
- COSRanger Monitoring Metrics
- Doris Monitoring Metrics
- Kylin Monitoring Metrics
- Zeppelin Monitoring Metrics
- Oozie Monitoring Metrics
- Storm Monitoring Metrics
- Livy Monitoring Metrics
- Kyuubi Monitoring Metrics
- StarRocks Monitoring Metrics
- Kafka Monitoring Metrics
- Alarm Configurations
- Alarm Records
- EMR on TKE Operation Guide
- EMR Serverless HBase Operation Guide
- EMR Development Guide
- Hadoop Development Guide
- HDFS Common Operations
- HDFS Federation Management Development Guide
- HDFS Federation Management
- Submitting MapReduce Tasks
- Automatically Adding Task Nodes Without Assigning ApplicationMasters
- YARN Task Queue Management
- Practices on YARN Label Scheduling
- Hadoop Practical Tutorial
- Using API to Analyze Data in HDFS and COS
- Dumping YARN Job Logs to COS
- Spark Development Guide
- Hbase Development Guide
- Phoenix on Hbase Development Guide
- Hive Development Guide
- Presto Development Guide
- Sqoop Development Guide
- Hue Development Guide
- Oozie Development Guide
- Flume Development Guide
- Kerberos Development Guide
- Knox Development Guide
- Alluxio Development Guide
- Kylin Development Guide
- Livy Development Guide
- Kyuubi Development Guide
- Zeppelin Development Guide
- Hudi Development Guide
- Superset Development Guide
- Impala Development Guide
- Druid Development Guide
- TensorFlow Development Guide
- Kudu Development Guide
- Ranger Development Guide
- Kafka Development Guide
- Iceberg Development Guide
- StarRocks Development Guide
- Flink Development Guide
- Hadoop Development Guide
- Practical Tutorial
- API Documentation
- FAQs
- Service Level Agreement
- Contact Us
Flume Overview
Apache Flume is a highly available, distributed, and configurable tool/service that collects and aggregates large amounts of data such as logs and events from different sources. It is designed to collect data flows (e.g., log data) from various web servers and store them in centralized data storage system like HDFS and HBase.
Flume Architecture
A Flume event is defined as a unit of data flow. A Flume agent is a JVM process that contains the components required for completing a task. Among them, Source, Channel, and Sink are the core ones.
Source
It consumes an event passed to it by an external source (e.g., web servers or other sources) and save it to one or more channels.
Channel
Located between a source and a sink, a channel is used to cache incoming events. After the sink successfully sends the events to the channel at the next hop or the final destination, the events are removed from the channel.
Sink
A sink is responsible for transferring the events to the next hop or final destination and removing them from the channel upon transfer completion.
Instructions
Preparations
Create an EMR cluster. When creating the EMR cluster, you need to select the Flume component on the software configuration page.
Install Flume in the
/usr/local/service/flume path on the core and task nodes (CVM instances) of the EMR cluster. The installation path for master nodes is /usr/local/service/apps/.Configuring Flume
Go to the 
/usr/local/service/flume folder and create a file named example.conf.
# example.conf: A single-node Flume configuration# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources.r1.port = 44444# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1
Starting Flume
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Configuring a test sample
After successful configuration, you will see the Flume agent started previously printing to the terminal.
telnet localhost 44444Trying 127.0.0.1...Connected to localhost.localdomain (127.0.0.1).Escape character is '^]'.Hello world! <ENTER>OK
Yes
No
Was this page helpful?