tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
TDMQ for Apache Pulsar
    Documentation
    Download PDF

    Connecting to Prometheus

    Last updated: 2024-12-02 17:10:17
    Download PDF
      TDMQ for Apache Pulsar supports integrating pro cluster monitoring data into users' self-built Prometheus, which facilitates the observation of the TDMQ cluster's operations. It also allows for timely HPA of business workloads through Prometheus monitoring data, making overall online ops more automated.
      Advanced Usage:
      Using the standard Prometheus monitoring format, you can access data with your own Prometheus through our provided Exporter;
      For instance, configuring a Prometheus data source in Grafana or accessing Prometheus data with K8s for Workload HPA, etc.
      Note：
      If your professional cluster was created before March 25, 2024, enabling this feature requires a cluster upgrade. Contact us for assistance.

      User Guide

      1. Log in to the TDMQ for Apache Pulsar Pro Cluster console.
      2. In the left sidebar, choose Cluster Management, and click the target instance's ID to enter the instance basic information page.
      3. Click to enter the Monitoring Information Sheet page.
      4. In the top right corner, click the TMP Entry Access button to Access Monitoring Targets, and select the appropriate network type and network configuration.
      5. Click Submit to access a set of monitoring targets.
      6. Modify the configuration file prometheus.yml to add a node_exporter scraping task.
      job_name: prometheus-tdmq-pulsar
      honor_timestamps: true
      scrape_interval: 60s
      metrics_path: /tencent-cloud-metrics/
      scheme: http
      static_configs:
      - targets:
        - 10.x.x.x:10001
        - 10.x.x.x:10002
      When honor_timestamps is set to true, Prometheus will use the timestamps provided by the Exporter for its metrics rather than using the timestamps at which the Prometheus server receives these metrics.
      scrape_interval, the frequency of scraping monitoring metrics data. Currently at minute-level.
      metrics_path, the path to access monitoring metrics. Set it to /tencent-cloud-metrics/.
      scheme, the protocol for accessing resources in the configuration. Currently it only supports http.
      7. Access the corresponding visualization interface to view the configured monitoring metrics.
      Note:
      1. This feature provides monitoring data at the cluster level, Topic level, and subscription level for pro clusters, aligning perfectly with the metric items on the console. However, unlike the console, where the monitoring data is pre-aggregated, the data through Prometheus is post-aggregated and requires you to manually aggregate and display it on the Grafana dashboard.
      2. Due to the different paths of monitoring data collection and different aggregation logic, the data captured by this feature may differ in specific values from the data displayed on the user console.
      3. This feature is only supported by pro clusters.

      Metric Description

      Cluster Level

      Metric
      Metric Name
      Cluster message production rate
      pulsar_caculate_rate_in
      Cluster message consumption rate
      pulsar_caculate_rate_out
      Cluster message storage size
      pulsar_storage_size
      Cluster production bandwidth peak
      pulsar_throughput_in
      Cluster consumption bandwidth peak
      pulsar_throughput_out

      Topic Level

      Metric
      Metric Name
      Message production rate
      pulsar_caculate_rate_in
      Message consumption rate
      pulsar_caculate_rate_out
      Message production throughput
      pulsar_throughput_in
      Message consumption throughput
      pulsar_throughput_out
      Producer count
      pulsar_producers_count
      Consumer count
      pulsar_consumers_count
      Message storage size
      pulsar_storage_size
      Message backlog count
      pulsar_msg_backlog
      Message backlog size
      pulsar_storage_backlog_size
      Producer rate limiting count
      pulsar_publish_rate_limit_times
      Filtered message count
      pulsar_tag_filter_rejected_msg_rate

      Subscription Level

      Metric
      Metric Name
      Message backlog entries
      pulsar_subscription_back_log
      Message consumption rate
      pulsar_subscription_msg_rate_out
      Message consumption throughput
      pulsar_subscription_msg_throughput_out
      Unacknowledged message count
      pulsar_subscription_unacked_messages
      Consumer count
      pulsar_subscription_consumers_count
      Delayed message count
      pulsar_subscription_delayed
      Message deletion rate
      pulsar_subscription_msg_rate_expired
      Filtered message count
      pulsar_subscription_tag_filter_rejected_msg_rate
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy