Amazon SageMaker AI Gets Smarter Routing for Faster LLM Inference
Amazon SageMaker AI now supports KV cache-aware routing on real-time inference endpoints, letting you slash time-to-first-token for LLM workloads by intelligently routing requests to instances holding matching prompt cache. The feature works globally across all AWS regions and plays nicely with vLLM containers, LoRA adapters, and your existing OpenAI-compatible APIs—no extra costs attached.
source: [aws/whats-new]