tencent cloud

Feedback

Rollup Operations

Last updated: 2021-11-05 10:43:14

    Creating Rollup Task

    In scenarios with massive amounts of data, a business system can generate petabytes of data per day or even per hour. Time series data is generally massive in amount, time-sensitive, and trending. Therefore, systems using such data (e.g., monitoring or data analysis systems) often only need high-precision data in the most recent time period while downsampling historical data for storage. You can configure a rollup task to periodically aggregate historical data and save it into a new metric. A rollup task not only saves downsampled historical data, but also improves query performance and reduces storage costs. Note that a rollup task automatically creates child metrics inheriting all configurations of the parent metric according to base_metric, and the parent metric configurations will be overwritten if options is specified.

    1. Request address

    The address is the instance IP and port, such as 10.13.20.15:9200, which can be obtained in the console.

    2. Request path and method

    Path: /_rollup/${rollup_task_name}, where ${rollup_task_name} is the rollup task name.
    Method: PUT

    3. Request parameters

    None

    4. Request content

    Parameter Required Type Description
    base_metric Yes string Name of the metric (parent metric) depended on by the rollup task
    rollup_metric Yes string Name of the metric (child metric) generated by the rollup task
    base_rollup No string Rollup task depended on by the current rollup task. Before executing the current task, the system will check whether the dependent task in the corresponding time period has been completed
    query No string Query condition for data filtering, which consist of many elements and operations, such as name:host AND type:max OR region:gz
    group_by Yes Array Tag to be aggregated. Multiple tags can be included
    function Yes Map Aggregate name, mode, and fields. The fields can be only from the fields field in base_metric. If the fields field is empty, the rollup cannot be configured. The function can be sum, avg, min, max, set, any, first, last, percentiles, etc., such as {"cost_total":{"sum": {"field":"cost"}},"cpu_usage_avg":{ "avg": { "field":"cpu_usage"}}}
    interval Yes string Aggregate granularity, such as 1s, 5m, 1h, and 1d
    frequency No string Scheduling frequency, such as 5m, 1h, and 1d, which is the same as interval by default
    delay No string Execution delay. Generally, there should a certain delay for data write, such as 5m or 1h, in order to avoid data loss
    start_time No string Start time of periodic execution of the rollup task, which is the current time by default
    end_time No string Scheduling end time, which is the maximum timestamp value by default
    options No map rollup_metric options, which are the same as the options for metric creation

    5. Response content

    You need to judge whether a request is successful based on the error field. If the response content contains the error field, the request failed. For the error details, please see the error field description.

    6. Sample code for curl

    Request:

    curl -u root:le201909 -H 'Content-Type:application/json' -X PUT 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test -d'
    {
       "base_metric": %{base_metric_name},      
       "rollup_metric": %{rollup_metric_name},    
       "base_rollup": %{base_rollup_name},        
       "query" : "name:host AND type:max",   
       "group_by": ["host"],   
       "function": {
           "cost_total":  {
               "sum": { 
                   "field": "cost"
               }
           },
           "cpu_usage_avg": {
               "avg": {
                   "field": "cpu_usage"
               }
           },
           "value": {   
             "percentiles": {      
               "field": "value",
               "percents": [
                 95
               ]
             }
           },
           "metricName": {
             "set": {              
               "value": "cpu_usage"
             }
           },
           "appid": {
             "any": {             
               "field": "appid"
             }
           },
            "first_value": { 
             "first": {           
               "field": "value"
             }
           },
            "last_value": {
             "last": {             
               "field": "value"
             }
           }
       },
       "interval": "1m",           
       "frequency": "5m",          
       "delay": "1m",             
       "start_time": "1502892000",
       "end_time": "2147483647",
       "options": {
           "expire_day": 365
       }
    }
    '
    

    Response:

    {
       "acknowledged": true,
       "message": "create rollup success"
    }
    

    Getting Names of All Rollup Tasks

    1. Request address

    The address is the instance IP and port, such as 10.13.20.15:9200, which can be obtained in the console.

    2. Request path and method

    Path: /_rollups
    Method: GET

    3. Request parameters

    None

    4. Request content

    None

    5. Response content

    You need to judge whether a request is successful based on the error field. If the response content contains the error field, the request failed. For the error details, please see the error field description.

    6. Sample code for curl

    Request:
    curl -u root:le201909 -H 'Content-Type:application/json' -X GET 172.16.345.14:9201/_rollups

    Response:

    {
       "result": 
    {
        "rollups": 
        [
            "rollup_jgq_6",
            "rollup_jgq_60"
        ]
       },
       "status": 200
    }
    

    Getting Rollup Task Details

    1. Request address

    The address is the instance IP and port, such as 10.13.20.15:9200.

    2. Request path and method

    Path: /_rollup/${rollup_task_name}, where ${rollup_task_name} is the rollup task name.
    Method: GET

    3. Request parameters

    You can specify the v parameter to view the specific rollup progress. @last_end_time in the response structure is the latest rollup progress.

    4. Request content

    None

    5. Response content

    You need to judge whether a request is successful based on the error field. If the response content contains the error field, the request failed. For the error details, please see the error field description.

    6. Sample code for curl

    Request:
    curl -u root:le201909 -H 'Content-Type:application/json' -X GET 172.16.345.14:9201/_rollup/rollup_jgq_6?v

    Response:

    {
     "result": {
       "rollup_jgq_6": {
       "base_metric": "cvm_device-300",
          "rollup_metric": "cvm_device-86400",
          "query": "metricName:cpu_usage AND statType:max",
          "group_by": [
            "vm_uuid"
          ],
          "function": {
            "value": {
              "percentiles": {
                "field": "value",
                "percents": [
                  95
                ]
              }
            },
            "metricName": {
              "set": {
                "value": "cpu_usage"
              }
            },
            "appid": {
              "any": {
                "field": "appid"
              }
            }
          },
          "interval": "1d",
          "delay": "5m",
          "options": {
            "expire_day": 186
          },
          "frequency": "1d",
          "start_time": 1534003200,
          "end_time": 2147483647,
          "@state": "running",             // Running status
          "@timestamp": 1550766085000,     // Rollup task information update time point
          "@last_end_time": 1550764800     // Rollup task end time point for proper execution
        }
     },
     "status": 200
    }
    

    Deleting Rollup Task

    1. Request address

    The address is the instance IP and port, such as 10.13.20.15:9200.

    2. Request path and method

    Path: /_rollup/${rollup_task_name}, where ${rollup_task_name} is the rollup task name.
    Method: DELETE

    3. Request parameters

    None

    4. Request content

    None

    5. Response content

    You need to judge whether a request is successful based on the error field. If the response content contains the error field, the request failed. For the error details, please see the error field description.

    6. Sample code for curl

    Request:
    curl -u root:le201909 -H 'Content-Type:application/json' -X DELETE 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test

    Response:

    {
       "acknowledged": true,
       "message": "delete rollup success"
    }
    

    Updating Rollup Task

    1. Request address

    The address is the instance IP and port, such as 10.13.20.15:9200.

    2. Request path and method

    Path: /_rollup/${rollup_task_name}/update, where ${rollup_task_name} is the rollup task name.
    Method: POST

    3. Request parameters

    None

    4. Request content

    Parameter Required Type Description
    state Yes string Valid values: running, pause
    start_time No string Start time of periodic execution of the rollup task, which is the current time by default
    end_time No string Scheduling end time, which is the maximum timestamp value by default
    options No map Aggregate options, which are the same as the options for metric creation

    5. Response content

    You need to judge whether a request is successful based on the error field. If the response content contains the error field, the request failed. For the error details, please see the error field description.

    6. Sample code for curl

    Request:

    curl -u root:le201909 -H 'Content-Type:application/json' -X POST 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test/update -d'
    {
       "state":"running",
       "start_time": "1511918989",
       "end_time": "1512019765",
       "options": 
    {
           "expire_day": 365
       }
    }'
    

    Response:

    {
       "acknowledged": true,
       "message": "update rollup success"
    }
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support