Scaling a Kafka Consumer From 4K to 25K Events per Second While Preserving Ordering
A Production Case Study in Batching, Ordering, and Offset Management
In one of our event-driven applications, the Kafka consumer was processing approximately 4,000 events per second, while the production readiness SLO required at least 8,000. The gap forced us to reconsider the consumer design rather than continue tuning the existing record-by-record processing model. We introduced batch processing, which increased throughput to approximately 25,000 events per second. However, improving throughput was only part of the problem. The new design also had to preserve event ordering, commit offsets safely, and maintain at-least-once delivery without silently losing events. This article explains the design decisions behind that redesign and the tradeoffs involved in improving performance without weakening processing guarantees.
The Scaling Constraint
As incoming traffic increased, the consumer could no longer keep up with the rate at which events were being produced, causing consumer lag to grow. The most direct scaling option was to...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
