- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Instance Management
- Creating Instance
- Naming with Consecutive Numeric Suffixes or Designated Pattern String
- Viewing Instance
- Upgrading Instance
- Downgrading Instance Configuration
- Terminating/Returning Instances
- Change from Pay-as-You-Go to Monthly Subscription
- Upgrading Instance Version
- Adding Routing Policy
- Public Network Bandwidth Management
- Connecting to Prometheus
- AZ Migration
- Setting Maintenance Time
- Setting Message Size
- Topic Management
- Consumer Group
- Monitoring and Alarms
- Smart Ops
- Permission Management
- Tag Management
- Querying Message
- Event Center
- Migration to Cloud
- Data Compression
- Instance Management
- CKafka Connector
- Practical Tutorial
- Practical Tutorial of CKafka Client
- Connector Practical Tutorial
- Connecting Flink to CKafka
- Connecting Schema Registry to CKafka
- Connecting Spark Streaming to CKafka
- Connecting Flume to CKafka
- Connecting Kafka Connect to CKafka
- Connecting Storm to CKafka
- Connecting Logstash to CKafka
- Connecting Filebeat to CKafka
- Multi-AZ Deployment
- Log Access
- Replacing Supportive Route (Old)
- Practice Tutorial for Cluster Bandwidth in High CPU Utilization Scenarios
- Practice Tutorial for Cluster Capacity Planning
- Troubleshooting
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- DataHub APIs
- ACL APIs
- Topic APIs
- DescribeTopicProduceConnection
- BatchModifyGroupOffsets
- BatchModifyTopicAttributes
- CreateConsumer
- CreateDatahubTopic
- CreatePartition
- CreateTopic
- CreateTopicIpWhiteList
- DeleteTopic
- DeleteTopicIpWhiteList
- DescribeDatahubTopic
- DescribeTopic
- DescribeTopicAttributes
- DescribeTopicDetail
- DescribeTopicSubscribeGroup
- FetchMessageByOffset
- FetchMessageListByOffset
- ModifyDatahubTopic
- ModifyTopicAttributes
- DescribeTopicSyncReplica
- Instance APIs
- Route APIs
- Other APIs
- Data Types
- Error Codes
- SDK Documentation
- General References
- Conducting Production and Consumption Pressure Testing on CKafka
- Configuration Guide for Common Parameters in CKafka
- Connecting to Legacy Self-Built Kafka
- Suggestions for CKafka Version Selection
- CKafka Data Reliability Description
- Connector
- FAQs
- Service Level Agreement
- Contact Us
- Glossary
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Instance Management
- Creating Instance
- Naming with Consecutive Numeric Suffixes or Designated Pattern String
- Viewing Instance
- Upgrading Instance
- Downgrading Instance Configuration
- Terminating/Returning Instances
- Change from Pay-as-You-Go to Monthly Subscription
- Upgrading Instance Version
- Adding Routing Policy
- Public Network Bandwidth Management
- Connecting to Prometheus
- AZ Migration
- Setting Maintenance Time
- Setting Message Size
- Topic Management
- Consumer Group
- Monitoring and Alarms
- Smart Ops
- Permission Management
- Tag Management
- Querying Message
- Event Center
- Migration to Cloud
- Data Compression
- Instance Management
- CKafka Connector
- Practical Tutorial
- Practical Tutorial of CKafka Client
- Connector Practical Tutorial
- Connecting Flink to CKafka
- Connecting Schema Registry to CKafka
- Connecting Spark Streaming to CKafka
- Connecting Flume to CKafka
- Connecting Kafka Connect to CKafka
- Connecting Storm to CKafka
- Connecting Logstash to CKafka
- Connecting Filebeat to CKafka
- Multi-AZ Deployment
- Log Access
- Replacing Supportive Route (Old)
- Practice Tutorial for Cluster Bandwidth in High CPU Utilization Scenarios
- Practice Tutorial for Cluster Capacity Planning
- Troubleshooting
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- DataHub APIs
- ACL APIs
- Topic APIs
- DescribeTopicProduceConnection
- BatchModifyGroupOffsets
- BatchModifyTopicAttributes
- CreateConsumer
- CreateDatahubTopic
- CreatePartition
- CreateTopic
- CreateTopicIpWhiteList
- DeleteTopic
- DeleteTopicIpWhiteList
- DescribeDatahubTopic
- DescribeTopic
- DescribeTopicAttributes
- DescribeTopicDetail
- DescribeTopicSubscribeGroup
- FetchMessageByOffset
- FetchMessageListByOffset
- ModifyDatahubTopic
- ModifyTopicAttributes
- DescribeTopicSyncReplica
- Instance APIs
- Route APIs
- Other APIs
- Data Types
- Error Codes
- SDK Documentation
- General References
- Conducting Production and Consumption Pressure Testing on CKafka
- Configuration Guide for Common Parameters in CKafka
- Connecting to Legacy Self-Built Kafka
- Suggestions for CKafka Version Selection
- CKafka Data Reliability Description
- Connector
- FAQs
- Service Level Agreement
- Contact Us
- Glossary
Overview
With the development of technology, Serverless architectures and usage patterns offer users the dual benefits of technical sophistication and cost efficiency. On the one hand, commercial PaaS products provide strong technical capabilities to support system architecture design. On the other hand, the billing mode has been shifted from purchasing fixed specifications to flexible, pay-as-you-go pricing, allowing customers to pay based on actual storage utilization. Additionally, storage supports elastic scale-out, which helps customers improve resource utilization and reduce business costs.
For example, a customer purchasing a cluster might need to set a minimum storage size of 200 GB. However, in the initial stages of business, actual utilization may be less than 10%. As business volume grows over time, storage would need to be scaled out continuously without changing specifications.
Therefore, elastic storage is introduced in TDMQ for CKafka Pro Edition.
How It Works
The elastic storage in the Pro Edition is achieved through a combination of local storage and remote storage, using a tiered storage approach. This means that a small amount of hot data is stored on local cloud disks, while large amounts of cold data are stored in remote storage.
Local Storage
Local storage supports write traffic and Tail-Read operations, delivering the same latency, availability, and consistency as these of the native Kafka.
If remote storage fails or experiences performance degradation, local storage supports elastic scale-out to continue providing read and write services.
Remote Storage
Remote storage supports Catch-Up read with a separation of hot and cold data.
On-demand usage with pay-as-you-go pricing.
Designed for future scalability, it supports multimodal and multi-medium storage.
This technical solution maintains write latency that is consistent with local storage. In cases of remote storage failures or slowdowns, it can fall back to local storage, dynamically scaling out with the support of a sutomatic operation system. Additionally, remote storage is relatively cost-effective, helping to reduce costs to some extent.
Tiered Storage Read and Write Process
Production Process
The main production process is similar to that of the native Kafka, where data written to cloud disk is asynchronously synchronized to remote storage on COS.
Consumption Process
The consumption process is also similar, comparing the user’s consumer offset to determine the data location. If the data is stored in local storage, it is prioritized and retrieved locally. If it is not available in local storage, the system retrieves it in real time from remote storage, applying different download policies based on specific read requirements to facilitate message consumption.
For more information on data lifecycle management, offset constraints, segment status machines, and tiered storage architecture, see The Practice and Evolution of Kafka Hierarchical Storage in Tencent Cloud.
Feature Overview
Usage Instructions
When you purchase a cluster in the console:
Specification type: Select the Pro edition.
Storage type: Select Elastic Storage.
After other options are confirmed, click Create.
Billing Overview
Billing is based on the storage space occupied by message storage and the duration of storage.
Billing method: Pay-as-you-go (postpaid), with the unit of XX CNY/GB/hour.
Billing granularity: By hour, with any period less than one hour rounded up to a full hour. Billing is based on the maximum disk capacity used within each hour.
Monitoring Metrics
The Pro Edition - Elastic Storage instance provides three metrics to help you monitor storage usage.
Instance level: Instance disk usage (MB), representing the storage usage of the current instance, including replica consumption.
Topic level: Total disk usage for Topics (MB), representing the storage usage of the current Topic, excluding replica consumption.
Partition level: Total disk usage for Partitions (MB), representing the storage usage under the current Topic-partition, excluding replica consumption.
Use Limits
1. Engine version
Currently, elastic storage is only supported for the Pro Edition Kafka 2.8.1 engine version.
2. Region
This configuration is only available in Beijing, Shanghai, Guangzhou, Singapore, and Hong Kong (China).
3. Availability
This feature is on an allowlist basis. If you want to use it, contact an after-sales architect or contact us.
Yes
No
Was this page helpful?