tencent cloud

All product documents
TDMQ for CKafka
DocumentationTDMQ for CKafkaPurchase GuideProfessional Edition - Elastic Storage
Professional Edition - Elastic Storage
Download PDF
Last updated: 2024-11-07 14:57:01
Professional Edition - Elastic Storage
Last updated: 2024-11-07 14:57:01
Download PDF

Overview

With the development of technology, Serverless architectures and usage patterns offer users the dual benefits of technical sophistication and cost efficiency. On the one hand, commercial PaaS products provide strong technical capabilities to support system architecture design. On the other hand, the billing mode has been shifted from purchasing fixed specifications to flexible, pay-as-you-go pricing, allowing customers to pay based on actual storage utilization. Additionally, storage supports elastic scale-out, which helps customers improve resource utilization and reduce business costs.
For example, a customer purchasing a cluster might need to set a minimum storage size of 200 GB. However, in the initial stages of business, actual utilization may be less than 10%. As business volume grows over time, storage would need to be scaled out continuously without changing specifications.
Therefore, elastic storage is introduced in TDMQ for CKafka Pro Edition.

How It Works

The elastic storage in the Pro Edition is achieved through a combination of local storage and remote storage, using a tiered storage approach. This means that a small amount of hot data is stored on local cloud disks, while large amounts of cold data are stored in remote storage.


Local Storage

Local storage supports write traffic and Tail-Read operations, delivering the same latency, availability, and consistency as these of the native Kafka.
If remote storage fails or experiences performance degradation, local storage supports elastic scale-out to continue providing read and write services.

Remote Storage

Remote storage supports Catch-Up read with a separation of hot and cold data.
On-demand usage with pay-as-you-go pricing.
Designed for future scalability, it supports multimodal and multi-medium storage.
This technical solution maintains write latency that is consistent with local storage. In cases of remote storage failures or slowdowns, it can fall back to local storage, dynamically scaling out with the support of a sutomatic operation system. Additionally, remote storage is relatively cost-effective, helping to reduce costs to some extent.

Tiered Storage Read and Write Process

Production Process

The main production process is similar to that of the native Kafka, where data written to cloud disk is asynchronously synchronized to remote storage on COS.


Consumption Process

The consumption process is also similar, comparing the user’s consumer offset to determine the data location. If the data is stored in local storage, it is prioritized and retrieved locally. If it is not available in local storage, the system retrieves it in real time from remote storage, applying different download policies based on specific read requirements to facilitate message consumption.

For more information on data lifecycle management, offset constraints, segment status machines, and tiered storage architecture, see The Practice and Evolution of Kafka Hierarchical Storage in Tencent Cloud.

Feature Overview

Usage Instructions

When you purchase a cluster in the console:
Specification type: Select the Pro edition.
Storage type: Select Elastic Storage.
After other options are confirmed, click Create.

Billing Overview

Billing is based on the storage space occupied by message storage and the duration of storage.
Billing method: Pay-as-you-go (postpaid), with the unit of XX CNY/GB/hour.
Billing granularity: By hour, with any period less than one hour rounded up to a full hour. Billing is based on the maximum disk capacity used within each hour.

Monitoring Metrics

The Pro Edition - Elastic Storage instance provides three metrics to help you monitor storage usage.
Instance level: Instance disk usage (MB), representing the storage usage of the current instance, including replica consumption.
Topic level: Total disk usage for Topics (MB), representing the storage usage of the current Topic, excluding replica consumption.
Partition level: Total disk usage for Partitions (MB), representing the storage usage under the current Topic-partition, excluding replica consumption.

Use Limits

1. Engine version
Currently, elastic storage is only supported for the Pro Edition Kafka 2.8.1 engine version.
2. Region
This configuration is only available in Beijing, Shanghai, Guangzhou, Singapore, and Hong Kong (China).
3. Availability
This feature is on an allowlist basis. If you want to use it, contact an after-sales architect or contact us.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Contact Us
Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 available.

Submit a Ticket
7x24 Phone Support
Hong Kong, China
+852 800 906 020 (Toll Free)
United States
+1 844 606 0804 (Toll Free)
United Kingdom
+44 808 196 4551 (Toll Free)
Canada
+1 888 605 7930 (Toll Free)
Australia
+61 1300 986 386 (Toll Free)
EdgeOne hotline
+852 300 80699
More local hotlines coming soon