Failure to Delete Expired Messages Promptly

TDMQ for CKafka

Release Notes and Announcements

Release Notes

Broker Release Notes

Connector Release Notes

DataHub Release Notes

Announcement

Elastic Topic Feature Plan Termination Public Beta Announcement

TDMQ for CKafka Advanced Edition Launch Announcement

Product Introduction

Technical Principles

Consumer Group Status Description

Traffic Throttling Mechanism Description

Comparison with Apache Kafka

Use Limits

Regions and AZs

High Availability

Multi-AZ Deployment

Multi-Replica and Election Mechanism

Data High Reliability

Migrating Availability Zones

Purchase Guide

Product Specifications

Professional Edition - Elastic Storage

Billing Overview

Purchasing pay-as-you-go instances

Monthly Subscription

Converting Pay-As-You-Go Instances to Monthly Subscription

Getting Started

Obtaining Access Permission

Getting Access Authorization

Granting Operation-Level Permissions to Sub-Accounts

Granting Resource-Level Permissions to Sub-Accounts

Granting Tag-Level Permissions to Sub-Accounts

VPC Access

Step 1. Create an Instance

Step 2. Create a Topic

Step 3. Add a VPC Route

Step 4. Send/Receive Messages

Using SDK to Receive/Send Message (Recommended)

Running Kafka Client (Optional)

Access via Public Domain Name

Step 1. Create an Instance

Step 2. Add a Public Route

Step 3. Create a Topic

Step 4. Configure an ACL Policy

Step 5. Send/Receive Messages

Using SDK to Receive/Send Message (Recommended)

Running Kafka Client (Optional)

Development Guide

CKafka Transaction Management

CKafka Version Selection Suggestion

Common Parameter Configuration Description of CKafka

CKafka Data Compression

Integrating Legacy Self-Built Kafka

Operation Guide

Instance Management

Creating Instance

Naming with Consecutive Numeric Suffixes or Designated Pattern String

Viewing Instance

Upgrading Instances

Downgrading Instances

Terminating/Returning Instances

Change from Pay-as-You-Go to Monthly Subscription

Upgrading Instance Version

Adding Routing Policy

Connecting to Prometheus

AZ Migration

Setting Maintenance Time

Setting Message Size

Topic Management

Instance Topic

Setting Topic Traffic Throttling Rule

Elastic Topic

Topic Management

Subscription Management

Consumer Group

Consumer Group Overview

Creating Consumer Group

Querying Consumer Group

Deleting Consumer Group

Setting Offset

Automatically Resetting Offset

Monitoring and Alarms

Viewing Monitoring Data

Querying Advanced Monitoring (Pro Edition)

Configuring Alarms

Suggestions for Alarm Configuration

Viewing Advanced Ops Features (Pro Edition)

Smart Ops

Elastic Scaling

Disk Utilization Policy

Dynamic Partition Processing

Elastic Bandwidth

Permission Management

Configuring ACL Policy

Migration to Cloud

Migration Scheme Overview

Step 1. Purchasing CKafka Instance

Step 2. Migrating Topic to Cloud

Step 3. Migrating Data to Cloud

Scheme 1: Single-Producer Dual-Consumer Migration

Scheme 2: Single-Producer Single-Consumer Migration

Scheme 3: Migration with MirrorMaker

Scheme 4: Migrating Unconsumed Data

Data Compression

CKafka Connector

Introduction

Connection Management

Task Management

Creating Data Access Task

Creating Data Distribution Task

Data Target

Data Distribution to TDW

Simple Data Processing

Data Conversion

Filter Rule Description

Task Management

Schema Management

Event Center

Practical Tutorial

Troubleshooting

Topic Failures

Topic Creation Failure

No Topic Monitoring Data

Linkage Failure of Other Tencent Cloud Services Caused by Topic ACL Policy Configuration

Existence of Partition Message Heap

Consumer Group Failures

No Consumer Group Details Displayed

Consumer Group Constantly in PreparingRebalance Status

Client Failures

Common Client Errors and Solutions

Blockage of Messages Produced by Client

Client’s Failure to Consume Messages

Sarama Client

Message Failures

Consumption Data Exception

Failure to Delete Expired Messages Promptly

Slow Consumption Speed

Warning Displayed for Message Heap

Error Persistence After a Period of Production

API Documentation

Making API Requests

DataHub APIs

DescribeDatahubTopics

ACL APIs

Topic APIs

Instance APIs

CreatePostPaidInstance

DescribeInstanceAttributes

DescribeInstancesDetail

InquireCkafkaPrice

ModifyGroupOffsets

ModifyInstanceAttributes

Route APIs

Other APIs

DeleteRouteTriggerTime

DescribeAppInfo

DescribeCkafkaZone

DescribeConsumerGroup

SDK Documentation

SDK for Java

VPC Access Through SASL_SCRAM

Public Network Access Through SASL_PLAINTEXT

Access Through SASL_SSL

SDK for Python

VPC Access

Public Network Access Through SASL_PLAINTEXT

Public Network Access Through SASL_SSL

SDK for Go

VPC Access

Public Network Access Through SASL_PLAINTEXT

SDK for PHP

VPC Access

Public Network Access Through SASL_PLAINTEXT

SDK for C++

VPC Access

Public Network Access Through SASL_PLAINTEXT

SDK for Node.js

VPC Access

Public Network Access Through SASL_PLAINTEXT

SDK for Connector

Data Reporting SDK

Elastic Topic Message Sending and Receiving

Connecting Filebeats to CKafka

Connecting Logstash to CKafka

General References

Conducting Production and Consumption Pressure Testing on CKafka

Configuration Guide for Common Parameters in CKafka

Connecting to Legacy Self-Built Kafka

Suggestions for CKafka Version Selection

CKafka Data Reliability Description

Connector

Database Change Subscription

MongoDB Data Subscription

MySQL Data Subscription

PostgreSQL Data Subscription

Official Format Description for MySQL Subscription Messages

Canal Format of MySQL Subscription Message

User Permission Settings Reference for PostgreSQL Subscription by Connector

Data Processing

Data Processing Rule Description

Regular Expression Extraction

JSONPath Description

Self-Built Cluster Connection Instructions (CLB Method)

Authorization Instructions for Access to CLS and COS Services Through Connectors

What Is a Signaling Table

FAQs

Connector

Data Reporting Issues

Data Processing Issues

Data Dump

Data Subscription Issues

Service Level Agreement

Glossary

DocumentationTDMQ for CKafkaTroubleshootingMessage FailuresFailure to Delete Expired Messages Promptly

Failure to Delete Expired Messages Promptly

Download PDF

Last updated: 2024-01-09 14:57:56

Failure to Delete Expired Messages Promptly

Last updated: 2024-01-09 14:57:56

Download PDF

Issue Description
Expired messages are not deleted promptly.
Possible Causes
The message deletion mechanism of Kafka may cause a problem where expired messages are not deleted promptly in certain business scenarios. You may feel confused if you are unfamiliar with this mechanism. For example, the message timestamps in partition 0 and partition 7 are obviously different, but the expired messages in partition 0 are not deleted promptly.
Kafka Message Deletion Mechanism
Kafka data is stored in three dimensions: topic, partition, and data segment. The message data deletion conditions are as follows:
Message data is deleted in the unit of data segment based on the retention period.
Currently, the maximum size of a data segment is set to 1 GB. After a data segment reaches 1 GB in size, a new segment will be generated, and so on.
Only after all messages in a data segment expire will the segment be deleted.
If a message in a data segment is still within the retention period, such as the last row of a segment file, then the file will not be deleted.
For some reasons, messages are written unevenly and concentrated in a certain partition (such as partition 7), while some other partitions (such as partition 0) have little data. In this case, the size of the data segment in partition 0 does not reach 1 GB, so no new segments are generated; however, there is data in the entire data segment within the retention period. Therefore, messages in partition 0 will not be deleted.

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

tencent cloud

New User Offers

Next-Generation CDN：EdgeOne

Elasticsearch Service free trial

Free Tier

Tencent Cloud Startup Program

Special Offers

Lighthouse Special Offers

Cloud Object Storage Special Offers

Featured Products

New Products

Education

Tencent Cloud Online Education Solutions

Gaming

Gaming Solution

Game Media Solutions

E-commerce

E-commerce retail solutions

Audio & Video

Audio/Video Solution

LVB Recording Solution

Interactive Classroom Solution

Interactive Live Streaming Solution

Audio Chat Social Networking Solution

Financial Services

Financial Services Solution

Compute

Cloud Virtual Machine

Auto Scaling

Batch Compute

CVM Dedicated Host

Database

TencentDB for MySQL

TencentDB for Redis®

TencentDB for CTSDB

TDSQL for MySQL

Data Transfer Service

TencentDB for MongoDB

TencentDB for PostgreSQL

TencentDB for SQL Server

TencentDB for TcaplusDB

Video Service

Cloud Streaming Services

Video on Demand

Media Processing Service

Cloud Application Rendering

Cloud Contact Center

Game Multimedia Engine

Chat

Real-time Communication

Tencent Effect SDK

AI and Machine Learning

Image Creation Large Model

Face Fusion

eKYC

Optical Character Recognition

Video Creation Large Model

Industry Applications

Tencent HealthCare Omics Platform

Container and Middleware

TDMQ for CKafka

Serverless Cloud Function

Tencent Kubernetes Engine

Tencent Kubernetes Engine for Serverless

Networking

Cloud Load Balancer

Virtual Private Cloud

Direct Connect

Cloud Connect Network

NAT Gateway

VPN Connection

Bandwidth Package

Anycast Internet Acceleration

Elastic Network Interface

Flow Logs

Global Application Acceleration Platform

Security

Captcha

Cloud Workload Protection Platform

Data Security Governance Center