bartek@aws: ~/news
$ whoami
$ AWS Architect · DevOps · Cloud
Tuesday, November 17, 2026

Amazon SageMaker AI Gets Smarter About Caching Your Prompts

Amazon SageMaker AI real-time endpoints now feature KV cache-aware routing, which intelligently directs each request to whichever instance has already cached the longest part of your prompt. This cuts down latency and boosts efficiency for LLM inference—you'll learn when to use it versus prefix-aware routing and how to set it up on your endpoints.

source: [aws/machine-learning-blog]