Prefill/Decode Disaggregation Can Make Tail Latency Worse
Prefill/decode disaggregation has the elegance of a clean architecture diagram.
Prompt processing is compute-heavy. Token generation is memory-bandwidth-heavy. Put them on separate GPU pools, scale each phase independently, and stop them from interfering with each other.
That logic is sound.
Then a burst arrives. Prefill queues grow. Decode GPUs have spare compute but cannot begin because KV cache has not reached them. The network carries large long-context state. Time to first token rises while each pool looks locally “correct.”
Disaggregation did not remove the bottleneck. It distributed it across queues and a data transfer.
Decompose TTFT After the Split
Before disaggregation:
TTFT = scheduler queue + prefill + first decode step
After disaggregation:
TTFT = routing + prefill queue + prefill execution + KV transfer queue + KV serialization/transfer + decode admission queue + first decode step
Phase isolation can reduce contention while increasing path length.
Instrument every component with...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
