Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference | Amazon Web Services
When you build an application on top of a large language model (LLM), the prompt you send to the model typically has two parts. There’s a fixed part that sets up context (instructions, reference documents, conversation history) and a variable part that contains the actual user input. Take a customer service bot as an example. Each request starts with the same block of text: “You are a support agent for AnyCompany. Here are our policies…” followed by whatever the customer typed. The instructions at the top might be 3,000 tokens. The customer’s question at the bottom might be 50 tokens.
This means that across hundreds or thousands of requests, your model is processing that same 3,000-token beginning over and over again.
LLM serving frameworks like vLLM and TensorRT-LLM have a solution for this. They cache the computed key-value (KV) pairs for prompt prefixes that have been seen before. When the...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE
