tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
TDMQ for CKafka
    Documentation
    Download PDF

    Sarama Client

    Last updated: 2024-11-07 15:37:11
    Download PDF

      Issue Description

      Sarama is a Kafka client written in Go with a high message throughput.
      After you actively increase the number of CKafka partitions due to a performance bottleneck, Sarama may be unable to perceive the partition rebalance, causing a failure to produce/consume messages in the new partitions.

      Common Causes

      After CKafka rebalances partitions for various reasons, it will take about 10 minutes for Sarama to perceive the partition changes and then pull the metadata of the current topic to parse the latest partition data. As the pull is time-consuming, it may be regarded as a failure sometimes.
      Note：
      By default, sarama.Metadata.RefreshFrequency of Sarama is set to 10 minutes. Therefore, it needs to take up to 20 minutes to perceive a partition update in the worst case. If you want to speed up the partition update perception, you can reduce the value of this parameter. For example, if you set it to 10s, the update can be perceived within 20s at most.
      In addition, the CKafka team has found occasional failures of Sarama to pull the latest partition information, causing message retention and random exception reporting.
      This problem has been repeatedly reported and fixed in the Sarama community. It occurs less frequently on the latest version but still exists.

      Solutions

      If you are sensitive to rebalance and use the Go technology stack, we recommend you migrate to Confluent-Kafka-Go, a client maintained by Confluent™.

      Comparison of Common Go Clients

      Go Client
      Strengths
      Limitations
      Sarama
      It is very active in the community and widely used. Questions raised in the community are answered and solved quickly.
      It performs well. As it is written in native Go, it supports async and high-concurrency operations.
      It is moderately stable. It may experience unknown errors after the number of partitions is increased and all partitions are rebalanced.
      Confluent-Kafka-go (recommended)
      It is very stable. As it actually provides a layer of encapsulation for librdkafka, and librdkafka has been running for many years in multiple languages, it can be reliable enough.
      It performs well. As it is implemented in C++ at the underlying layer, it requires fewer resources and computes faster.
      As it imports a C++ library, the Go compiler needs additional compilation configurations, which add compilation dependencies and make compilation more complex.
      In addition, as the logic for C++ library implementation varies by compilation environment, importing the librdkafka library may affect the cross-compilation of Go projects.
      kafka-go
      It provides top-level APIs and exposes underlying APIs, which can be called more flexibly to configure the Kafka client.
      It is easy to use since it requires less code to produce and consume messages and has many default configuration items.
      It underperforms the above two clients and may be unable to sustain a high concurrency.
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy