The Trade-Offs of Adaptive Key Salting in Kafka
At 2:14 AM, nine of your ten Kafka consumers are idle. The tenth is pinned at 100% CPU, its lag climbing by the second, and the coordinator has started evicting it for missing max.poll.interval.ms. Every eviction triggers a rebalance. Every rebalance stops the other nine consumers cold for the duration. You didn't lose a broker. You didn't lose a data center. You partitioned by user_id.
I've operated production streaming systems where this exact failure shows up under load. Not because anyone wrote bad code, but because hash(user_id) % num_partitions encodes an assumption that almost never holds: that traffic distributes uniformly across keys. It doesn't. It follows a power law, and the gap between "textbook Kafka" and "Kafka at 99th-percentile traffic" lives entirely in that gap.
The Zipfian Lie
Partitioning tutorials treat key selection as a hash-modulo exercise: pick a key with decent cardinality, feed it to murmur2, done. That holds...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
