tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Log Service
    Documentation
    Download PDF

    Metric Pre-aggregation

    Last updated: 2024-09-20 17:48:27
    Download PDF
      When large volumes of metric data are monitored or complex queries are executed, real-time queries can become slow, and writing query statements may become challenging. To address this, the metric storage system supports a pre-aggregation feature. Pre-aggregation allows for the pre-computation of complex queries (using PromQL) and stores the results as new metrics. These precomputed data can be directly used in subsequent queries and alarms, reducing real-time computation costs, improving query performance, and simplifying and streamlining monitoring configuration. Pre-aggregation is compatible with Prometheus Recording Rules, allowing native YAML configuration files to be directly imported.
      For example, the original metric prometheus_http_requests_total records the number of requests made to various Prometheus APIs with different response status codes.
      # HELP prometheus_http_requests_total Counter of HTTP requests.
      # TYPE prometheus_http_requests_total counter
      prometheus_http_requests_total{code="200",handler="/api/v1/label/:name/values"} 7
      prometheus_http_requests_total{code="200",handler="/api/v1/query"} 19
      prometheus_http_requests_total{code="200",handler="/api/v1/query_range"} 27
      prometheus_http_requests_total{code="200",handler="/graph"} 11
      prometheus_http_requests_total{code="200",handler="/metrics"} 8929
      prometheus_http_requests_total{code="200",handler="/static/*filepath"} 52
      prometheus_http_requests_total{code="302",handler="/"} 1
      prometheus_http_requests_total{code="400",handler="/api/v1/query_range"} 6
      If you need to regularly query the total number of requests for each status code, you can use the following configuration to create a pre-aggregation task:
      groups:
        - name: example
          rules:
          - record: code:prometheus_http_requests_total:sum
            expr: sum by (code) (prometheus_http_requests_total)
      Among them:
      sum by (code) (prometheus_http_requests_total) is the metric query statement (PromQL), which calculates the sum of request counts grouped by status code.
      code:prometheus_http_requests_total:sum is the generated metric name, which can be customized. You can use this name in subsequent queries to directly retrieve the precomputed metric.

      Application Scenario

      Improve metric query performance: When some complex queries need to be executed frequently, pre-aggregation can be used to precompute and store these queries as new metrics. This can significantly enhance query performance and reduce query response time.
      Simplify query statements: Complex query statements can be lengthy and difficult to read. By using pre-aggregation, you can store the results of complex queries as simplified metrics. This makes subsequent queries easier to write and maintain.
      Store core metrics separately: All data in the metric topic is retained for the same duration and is automatically cleared after expiration. If you need to store a subset of core metrics for a longer period, you can use a pre-aggregation task to store this portion of data in another metric topic.

      Prerequisites

      A metric topic has already been created.

      Directions

      Creating Task

      1. Log in to the Cloud Log Service console.
      2. In the left sidebar, click Metric Topic.
      3. Click the metric topic ID/name for which you want to create a pre-aggregation task to enter the metric topic management page.
      4. Click the Metric Pre-aggregation tab to enter the pre-aggregation task list page, then click Creating Task. The main configuration items are described as follows:
      Configuration Item
      Description
      Service Log
      Saves the task running logs in the cls_service_log log topic, making it easier to monitor the task's operation status. This log topic is free, and it is recommended to enable it.
      Query Statement
      The PromQL statement to be executed. The pre-aggregation task will run this statement at scheduled intervals to retrieve the execution results.
      Indicator Name
      The result of the execution statement will be stored under this metric name, which can be used for subsequent data queries. It supports English letters, numbers, underscores, and colons, and should conform to the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*.
      Custom Dimension
      Adds dimensions to the metric. If there is a conflict between the custom dimension and the dimension names in the execution statement results, the custom dimension takes precedence.
      Scheduling cycle
      The execution interval for the pre-aggregation task, with the range from 1 to 1440 minutes. It is recommended to use a 1-minute interval.
      Advanced Settings
      Target Metric Topic: Specifies where the pre-aggregated metric data will be stored. By default, it is stored in the current topic. If you want to store this data separately (e.g., to set a different retention period for this portion of data), you can store it in another metric topic.
      Delayed Execution: Since there may be delays in metric data collection, you can set a delayed execution to ensure the data is fully collected before the pre-aggregation task runs. The default delay is 30 seconds.
      5. Once the configuration is complete, click Submit.

      Import Configuration File

      1. Log in to the Cloud Log Service console.
      2. In the left sidebar, click Metric Topic.
      3. Click the metric topic ID/name for which you want to create a pre-aggregation task to enter the metric topic management page.
      4. Click the Metric Pre-aggregation tab to enter the pre-aggregation task list page, then click Import Configuration File. The main configuration items are described as follows:
      
      Configuration Item
      
      Description
      Enabling Status
      Indicates whether the task needs to run. Tasks that are not running will not generate pre-aggregated result data.
      Service Log
      Saves the task running logs in the cls_service_log log topic, making it easier to monitor the task's operation status. This log topic is free, and it is recommended to enable it.
      Execution Interval
      The execution interval for the pre-aggregation task, with the range from 1 to 1440 minutes. It is recommended to use a 1-minute interval.
      YAML Configuration
      Compatible with Prometheus Recording Rule YAML. The interval only supports a range from 1 to 1440 minutes. For example:
      groups:
        - name: example
          rules:
          - record: code:prometheus_http_requests_total:sum
            expr: sum by (code) (prometheus_http_requests_total)
      Advanced Settings
      Target Metric Topic: Specifies where the pre-aggregated metric data will be stored. By default, it is stored in the current topic. If you want to store this data separately (e.g., to set a different retention period), you can store it in another metric topic.
      Delayed Execution: Since there may be delays in metric data collection, you can set a delayed execution to ensure the data is fully collected before the pre-aggregation task runs. The default delay is 30 seconds.
      5. Click Submit to save the configuration file. A pre-aggregation task will be automatically generated, with the task name consisting of the group name and record from the YAML file.
      Note:
      Pre-aggregation tasks automatically generated through a configuration file should be modified via the configuration file; direct modification is not supported.

      Manage the Configuration File

      1. Log in to the Cloud Log Service console.
      2. In the left sidebar, click Metric Topic.
      3. Click the metric topic ID/name for which you want to create a pre-aggregation task to enter the metric topic management page.
      4. Click the Metric Pre-aggregation tab to enter the pre-aggregation task list page, then click Manage the Configuration File.
      5. In the list, you can view the existing configuration files. In the Operation column, you can manage the configuration files:
      Edit: Edit the configuration file, with the configuration items consistent with Import Configuration File. After modification, the system will automatically adjust the pre-aggregation tasks based on the latest configuration file (deleting tasks that no longer exist in the file, adding new tasks, and updating existing tasks).
      Pause: Suspend the pre-aggregation tasks associated with the configuration file. During the suspension, these tasks will no longer be executed.
      Delete: Remove the pre-aggregation tasks associated with the configuration file.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy