TECH NEWS
Disaggregated prefill and decode for LLM inference on SageMaker HyperPod | Amazon Web Services
When prefill and decode share a GPU, long prompts stall token generation for every concurrent request. Disaggregated Prefill and Decode (DPD) removes this interference by running each phase on separate GPU pools connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA). Large language model (LLM) inference has