In scenarios with massive amounts of data, a business system can generate petabytes of data per day or even per hour. Time series data is generally massive in amount, time-sensitive, and trending. Therefore, systems using such data (e.g., monitoring or data analysis systems) often only need high-precision data in the most recent time period while downsampling historical data for storage. You can configure a rollup task to periodically aggregate historical data and save it into a new metric. A rollup task not only saves downsampled historical data, but also improves query performance and reduces storage costs. Note that a rollup task automatically creates child metrics inheriting all configurations of the parent metric according to base_metric
, and the parent metric configurations will be overwritten if options
is specified.
The address is the instance IP and port, such as 10.13.20.15:9200
, which can be obtained in the console.
Path: /_rollup/${rollup_task_name}
, where ${rollup_task_name}
is the rollup task name.
Method: PUT
None
Parameter | Required | Type | Description |
---|---|---|---|
base_metric | Yes | string | Name of the metric (parent metric) depended on by the rollup task |
rollup_metric | Yes | string | Name of the metric (child metric) generated by the rollup task |
base_rollup | No | string | Rollup task depended on by the current rollup task. Before executing the current task, the system will check whether the dependent task in the corresponding time period has been completed |
query | No | string | Query condition for data filtering, which consist of many elements and operations, such as name:host AND type:max OR region:gz |
group_by | Yes | Array | Tag to be aggregated. Multiple tags can be included |
function | Yes | Map | Aggregate name, mode, and fields. The fields can be only from the fields field in base_metric . If the fields field is empty, the rollup cannot be configured. The function can be sum , avg , min , max , set , any , first , last , percentiles , etc., such as {"cost_total":{"sum": {"field":"cost"}},"cpu_usage_avg":{ "avg": { "field":"cpu_usage"}}} |
interval | Yes | string | Aggregate granularity, such as 1s, 5m, 1h, and 1d |
frequency | No | string | Scheduling frequency, such as 5m, 1h, and 1d, which is the same as interval by default |
delay | No | string | Execution delay. Generally, there should a certain delay for data write, such as 5m or 1h, in order to avoid data loss |
start_time | No | string | Start time of periodic execution of the rollup task, which is the current time by default |
end_time | No | string | Scheduling end time, which is the maximum timestamp value by default |
options | No | map | rollup_metric options, which are the same as the options for metric creation |
You need to judge whether a request is successful based on the error
field. If the response content contains the error
field, the request failed. For the error details, please see the error
field description.
Request:
curl -u root:le201909 -H 'Content-Type:application/json' -X PUT 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test -d'
{
"base_metric": %{base_metric_name},
"rollup_metric": %{rollup_metric_name},
"base_rollup": %{base_rollup_name},
"query" : "name:host AND type:max",
"group_by": ["host"],
"function": {
"cost_total": {
"sum": {
"field": "cost"
}
},
"cpu_usage_avg": {
"avg": {
"field": "cpu_usage"
}
},
"value": {
"percentiles": {
"field": "value",
"percents": [
95
]
}
},
"metricName": {
"set": {
"value": "cpu_usage"
}
},
"appid": {
"any": {
"field": "appid"
}
},
"first_value": {
"first": {
"field": "value"
}
},
"last_value": {
"last": {
"field": "value"
}
}
},
"interval": "1m",
"frequency": "5m",
"delay": "1m",
"start_time": "1502892000",
"end_time": "2147483647",
"options": {
"expire_day": 365
}
}
'
Response:
{
"acknowledged": true,
"message": "create rollup success"
}
The address is the instance IP and port, such as 10.13.20.15:9200
, which can be obtained in the console.
Path: /_rollups
Method: GET
None
None
You need to judge whether a request is successful based on the error
field. If the response content contains the error
field, the request failed. For the error details, please see the error
field description.
Request:curl -u root:le201909 -H 'Content-Type:application/json' -X GET 172.16.345.14:9201/_rollups
Response:
{
"result":
{
"rollups":
[
"rollup_jgq_6",
"rollup_jgq_60"
]
},
"status": 200
}
The address is the instance IP and port, such as 10.13.20.15:9200
.
Path: /_rollup/${rollup_task_name}
, where ${rollup_task_name}
is the rollup task name.
Method: GET
You can specify the v
parameter to view the specific rollup progress. @last_end_time
in the response structure is the latest rollup progress.
None
You need to judge whether a request is successful based on the error
field. If the response content contains the error
field, the request failed. For the error details, please see the error
field description.
Request:curl -u root:le201909 -H 'Content-Type:application/json' -X GET 172.16.345.14:9201/_rollup/rollup_jgq_6?v
Response:
{
"result": {
"rollup_jgq_6": {
"base_metric": "cvm_device-300",
"rollup_metric": "cvm_device-86400",
"query": "metricName:cpu_usage AND statType:max",
"group_by": [
"vm_uuid"
],
"function": {
"value": {
"percentiles": {
"field": "value",
"percents": [
95
]
}
},
"metricName": {
"set": {
"value": "cpu_usage"
}
},
"appid": {
"any": {
"field": "appid"
}
}
},
"interval": "1d",
"delay": "5m",
"options": {
"expire_day": 186
},
"frequency": "1d",
"start_time": 1534003200,
"end_time": 2147483647,
"@state": "running", // Running status
"@timestamp": 1550766085000, // Rollup task information update time point
"@last_end_time": 1550764800 // Rollup task end time point for proper execution
}
},
"status": 200
}
The address is the instance IP and port, such as 10.13.20.15:9200
.
Path: /_rollup/${rollup_task_name}
, where ${rollup_task_name}
is the rollup task name.
Method: DELETE
None
None
You need to judge whether a request is successful based on the error
field. If the response content contains the error
field, the request failed. For the error details, please see the error
field description.
Request:curl -u root:le201909 -H 'Content-Type:application/json' -X DELETE 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test
Response:
{
"acknowledged": true,
"message": "delete rollup success"
}
The address is the instance IP and port, such as 10.13.20.15:9200
.
Path: /_rollup/${rollup_task_name}/update
, where ${rollup_task_name}
is the rollup task name.
Method: POST
None
Parameter | Required | Type | Description |
---|---|---|---|
state | Yes | string | Valid values: running, pause |
start_time | No | string | Start time of periodic execution of the rollup task, which is the current time by default |
end_time | No | string | Scheduling end time, which is the maximum timestamp value by default |
options | No | map | Aggregate options, which are the same as the options for metric creation |
You need to judge whether a request is successful based on the error
field. If the response content contains the error
field, the request failed. For the error details, please see the error
field description.
Request:
curl -u root:le201909 -H 'Content-Type:application/json' -X POST 172.16.345.14:9201/_rollup/ctsdb_rollup_task_test/update -d'
{
"state":"running",
"start_time": "1511918989",
"end_time": "1512019765",
"options":
{
"expire_day": 365
}
}'
Response:
{
"acknowledged": true,
"message": "update rollup success"
}
Was this page helpful?