Amazon SageMaker AI real-time endpoints now support KV cache aware routing | Amazon Web Services

https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22381-featured-image.png

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.

Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory,...

Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE

Read more

https://iprsoftwaremedia.com/219/files/202609/808e2a365dfc634a953afe10755a5580/6aba49193d63324837dde89f_39454f98-f1a6-4f6a-b3ab-43fe63dad6e/39454f98-f1a6-4f6a-b3ab-43fe63dad6e_2b3fbe3e-f53...

Nvidia reports Q3 revenue up 103% YoY to $115.8B, above $109.1B est., Data Center revenue up 110% to $107.6B, and net income up 105% to $65.3B

More: CNBC, Wall Street Journal, Washington Post, Wall Street Journal, PYMNTS.com, The Verge, Decrypt, Washington Examiner, JusttheNews.com, Gizmodo, TechRadar, HotAir, Bloomberg, Fortune, The National, Investopedia, Neowin, The Guardian, Australian Financial Review, Yahoo Finance, Sydney Morning Herald, CTech, Kotaku, TRT World, VideoCardz.com, Inc, Axios, 9to5Mac, AppleInsider, Hindustan Times,