The AI inference bottleneck just got obliterated. AWS has partnered with the llm-d team to launch disaggregated inference capabilities that fundamentally restructure how large language models process requests at enterprise scale. This isn’t another incremental optimization—it’s a complete architectural paradigm shift that separates the compute-intensive prefill phase from the memory-bound decode phase, unlocking massive performance gains and cost efficiencies.
The Problem: Traditional Inference Is Hitting a Wall
Agentic AI workflows are generating 10x more tokens than simple chatbot responses, creating exponential processing demands that bog down traditional inference systems. The root issue lies in how LLM inference operates through two distinct phases with completely different resource requirements.
The prefill phase processes entire input prompts in parallel to generate key-value (KV) cache entries—this is compute-bound work that needs raw processing power. The decode phase autoregressively generates tokens one at a time while accessing model weights and the growing KV cache—this is memory-bound work that demands high bandwidth.
Traditional deployments force both phases to share the same hardware, creating a fundamental mismatch. It’s like running a Formula 1 race and a freight train on the same track—neither can operate at peak efficiency.
The llm-d Solution: Separation of Concerns at Infrastructure Scale
Built on top of vLLM, llm-d introduces Kubernetes-native orchestration that treats inference as a distributed system problem rather than single-node execution. The framework provides three game-changing capabilities that directly address the scalability crisis.
Intelligent Cache-Aware Routing
The breakthrough here is maintaining visibility into cache state across serving replicas. When agentic workflows involve multi-turn conversations, traditional routing sends requests to random instances, negating the benefits of prefix caching entirely.
llm-d’s scheduler tracks which KV cache entries exist on which GPUs and routes requests accordingly. For workflows with high prefix reuse, this delivers dramatic improvements in both throughput and latency by ensuring requests hit servers that already hold relevant cached context.
True Prefill-Decode Disaggregation
This is where the architecture gets revolutionary. Instead of forcing compute-intensive prefill and memory-bandwidth-intensive decode to compete for the same resources, llm-d separates them entirely.
Prefill servers optimize for processing input prompts efficiently. Decode servers focus purely on low-latency token generation. A sidecar coordinates point-to-point KV cache transfers over high-speed interconnects like AWS Elastic Fabric Adapter (EFA), ensuring decode servers receive necessary cached context with minimal overhead.
The result? Dramatically improved time-to-first-token (TTFT) and overall throughput, especially for workloads with long prompts or large models.
“The AI chip conversation has been NVIDIA’s story for years. The inference layer is where that starts to change. AWS chose Cerebras for disaggregated inference because moving data between chips is the bottleneck. Cerebras eliminates it. When the biggest cloud platforms start routing around GPUs, that is a structural shift worth paying attention to.” — @wlthxyz
Historical Context: Why This Matters Now
This architectural shift echoes major infrastructure revolutions in computing history. The separation of compute and storage in cloud computing transformed how we think about scalability. The microservices movement disaggregated monolithic applications into specialized components.
Disaggregated inference follows the same pattern—breaking apart monolithic inference into specialized, optimizable components. Just as the transition from mainframes to distributed systems unlocked the internet era, disaggregated inference could unlock true agentic AI at scale.
Advanced Capabilities: Expert Parallelism and Tiered Caching
For Mixture-of-Experts (MoE) models like DeepSeek-R1 and Qwen3.5, llm-d provides wide expert parallelism that distributes experts horizontally across multiple nodes while maintaining performance. This addresses the complex parallelism and communication requirements that make MoE deployment challenging at scale.
Tiered prefix caching extends effective KV cache size beyond GPU memory limits by intelligently offloading cache entries. This creates a memory hierarchy optimized for inference patterns rather than general-purpose computing.
“Great to see @NVIDIA Dynamo 1.0 ship with native vLLM support! Disaggregated serving, agentic-aware routing, and topology-aware K8s scaling — exciting building blocks for production distributed inference. 🚀” — @vllm_project
Production Reality: AWS Integration and Benchmarking
The llm-d-aws container includes AWS-specific libraries like EFA and libfabric, plus integration with the NIXL library for multi-node disaggregated inference. Extensive benchmarking across multiple iterations ensures stable performance on Amazon SageMaker HyperPod and Amazon EKS.
This isn’t theoretical research—it’s production-ready infrastructure that enterprises can deploy today. The “well-lit paths” approach provides reference architectures for different performance and scalability goals, removing the guesswork from complex distributed deployments.
The Bigger Picture: Infrastructure for Agentic AI
As AI transitions from prototype to production, efficient inference becomes the gating factor for real-world deployment. Disaggregated inference doesn’t just improve performance—it fundamentally changes the economics of running agentic AI at scale.
By optimizing each phase of inference independently, organizations can right-size their infrastructure, reduce costs, and handle the variable demands of agentic workflows. This architectural foundation makes large-scale agentic AI economically viable for the first time.
The revolution isn’t just technical—it’s economic. Disaggregated inference could be the key that unlocks the full potential of agentic AI across industries, transforming AI from an experimental technology into critical infrastructure.