Overview
When using Kafka, you need to monitor its running status, such as cluster status and message heap. TMP provides an exporter to monitor Kafka and offers an out-of-the-box Grafana monitoring dashboard for it. This document describes how to deploy the Kafka exporter and integrate it with the alert feature.
Note:
For easier export installation and management, we recommend you use TKE for unified management. Prerequisites
You have created a TKE cluster in the region and VPC of your TMP instance and created a namespace for the cluster. You have located and integrated the target TKE cluster in the Integrate with TKE section of the target TMP instance in the TMP console. For more information, please see Agent Management. Directions
Deploying exporter
1. Log in to the TKE console. 2. Click the ID/name of the cluster whose access credential you want to get to enter the cluster management page.
3. On the left sidebar, select Workload > Deployment to enter the Deployment page.
4. On the Deployment management page, click Create and select the target namespace to deploy the service. You can create in the console. Here, YAML is used to deploy the exporter. Below is a sample YAML configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: kafka-exporter
name: kafak-exporter
namespace: kafka-demo
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kafka-exporter
template:
metadata:
labels:
k8s-app: kafka-exporter
spec:
containers:
- args:
- --kafka.server=x.x.x.x:9092
image: danielqsj/kafka-exporter:latest
imagePullPolicy: IfNotPresent
name: kafka-exporter
ports:
- containerPort: 9121
name: metric-port
securityContext:
privileged: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: qcloudregistrykey
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
Adding scrape task
1. Log in to the TMP console and select the target TMP instance to enter the management page. 2. Click a cluster ID in the TKE cluster list to enter the Integrate with TKE page.
3. In Scrape Configuration, add Pod Monitor
to define a Prometheus scrape task. Below is a sample YAML configuration:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: kafka-exporter
namespace: cm-prometheus
spec:
podMetricsEndpoints:
- interval: 30s
port: metric-port
path: /metrics
relabelings:
- action: replace
sourceLabels:
- instance
regex: (.*)
targetLabel: instance
replacement: 'ckafka-xxxxxx'
- action: replace
sourceLabels:
- instance
regex: (.*)
targetLabel: ip
replacement: '1.x.x.x'
namespaceSelector:
matchNames:
- kafka-demo
selector:
matchLabels:
k8s-app: kafka-exporter
Note:
As the exporter and Kafka are deployed on different servers, we recommend you use the Prometheus relabeling mechanism to add the Kafka instance information to the monitoring metrics so as to locate problems more easily.
1. Log in to the TMP console and select the target TMP instance to enter the management page. 2. Click Integration Center to enter the Integration Center page. Find Kafka monitoring, install the corresponding Grafana dashboard, and then you can enable the Kafka monitoring dashboard to view instance monitoring data as shown below:
Integrating with alert feature
1. Log in to the TMP console and select the target TMP instance to enter the management page. 2. Click Alerting Rule and add the corresponding alerting rules. For more information, please see Creating Alerting Rule.
Was this page helpful?