Apache Kafka, Purgatory, and Hierarchical Timing Wheels
Murali Krishna Vysyaraju (TOGAF Certified)
Assistant Vice President - Genpact
Apache Kafka has a data structure called the "request purgatory". The purgatory holds any request that hasn't yet met its criteria to succeed but also hasn't yet resulted in an error. The problem is “How can we efficiently keep track of tens of thousands of requests that are being asynchronously satisfied by other activity in the cluster?”
Kafka implements several request types that cannot immediately be answered with a response. Examples:
- A produce request with acks=all cannot be considered complete until all in-sync replicas have acknowledged the write and we can guarantee it will not be lost if the leader fails.
- A fetch request with min.bytes=1 won't be answered until there is at least one new byte of data for the consumer to consume. This allows a "long poll" so that the consumer need not busy wait checking for new data to arrive.
These requests are considered complete when either (a) the criteria they requested is complete or (b) some timeout occurs.
The number of these asynchronous operations in flight at any time scales with the number of connections, which, for Kafka, is often tens of thousands.
The request purgatory is designed for such a large scale request handling, but the old implementation had a number of deficiencies.
In this blog, I would like to explain the problem with the old implementation and how the new implementation solved it. I will also present benchmark results.
For Complete reference, Please refer below URL