Disaggregated prefill and decode for LLM inference on SageMaker HyperPod | Amazon Web Services
When prefill and decode share a GPU, long prompts stall token generation for every concurrent request. Disaggregated Prefill and Decode (DPD) removes this interference by running each phase on separate GPU pools connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA). Large language model (LLM) inference has two fundamentally different phases. Prefill is compute-bound. It processes the entire input prompt in parallel to generate the initial key-value (KV) cache. Decode is memory-bound. It generates one token at a time and requires substantial memory bandwidth to access model weights and the growing KV cache. By disaggregating these into specialized engines, you can assign different parallel strategies to each phase. With this separation, you can tune time to first token (TTFT) and inter-token latency (ITL) independently, control tail latency more reliably than chunked prefill tuning, and keep long-context prefills from blocking ongoing decode requests. vLLM improves single-node efficiency through...
Copyright of this story solely belongs to amazon.com. To see the full text click HERE
