On November 9, 2026, researchers from MIT, UC Berkeley, and Georgia Tech submitted the preprint ‘Evolving Infra: Agent-Adaptive LLM Serving at Scale’ (arXiv:2611.12304), introducing an open-source inference system that dynamically adapts compute allocation and key-value (KV) cache management to multi-agent workloads. By decoupling GPU compute allocation from memory tiering, the architecture eliminates the multi-minute phase-switch overheads and cache-thrashing penalties that plague conventional prefill-decode disaggregated serving clusters when hosting autonomous AI agents.
Overcoming Phase-Switch Delays and Compute Imbalances
Modern large language model (LLM) serving increasingly relies on prefill-decode (PD) disaggregation, in which separate GPU nodes are dedicated to processing prompt tokens (prefill) and generating autoregressive tokens (decode). While this separation prevents long prompts from stalling token generation in conventional single-turn chat, it introduces acute inefficiencies when serving autonomous agentic workloads. Agents execute tasks across iterative steps comprising model requests and external tool calls, creating dynamic, heterogeneous context growth with extended idle pauses between steps.
When the proportion of prefill-heavy and decode-heavy agent tasks shifts across a cluster, existing systems must reassign worker nodes between phases. In production frameworks like NVIDIA Dynamo v1.5.1, reallocating a node requires terminating and relaunching the worker process, incurring an average downtime of 681 to 754 seconds during which the node cannot process requests. Alternative in-place switching mechanisms, such as those evaluated in Arrow, avoid relaunch delays but sever access to previously cached KV data, forfeiting up to 55.5% of potential cache hits. In contrast, Evolving Infra introduces symmetric serving engines built upon SGLang that share a unified execution loop. This allows nodes to transition between prefill and decode roles in an average of 6.2 seconds—more than 100 times faster than Dynamo—while serving active requests continuously.
Decoupled Memory Pools and Agent-Aware Context Tracking
A central challenge in agentic inference is that traditional least-recently-used (LRU) cache eviction policies misjudge the value of stored context. When an agent pauses to execute an external code interpreter, query a database, or wait for an asynchronous sub-agent branch, an LRU policy frequently evicts its cached prefix due to inactivity. Conversely, when an agent harness compacts context or prunes historical turns, conventional caches retain invalidated tokens because they were accessed recently. This disconnect results in severe cache thrashing and redundant re-prefill computation across shared clusters.
To resolve this tension, the authors decoupled compute allocation from KV cache management through two coordinated mechanisms:
- Shared Host-Memory KV Pool: Each node pairs high-bandwidth GPU memory (L1) with host CPU memory (L2) caching four times as many tokens. A cross-node transfer protocol built on Mooncake RDMA transport enables any prefill or decode node to retrieve cached prefixes across the network, ensuring cached data remains accessible regardless of which phase a physical node currently executes.
- Cluster-Wide Agent-Aware Directory: A centralized directory tracks the active branch life cycle of each agent. By monitoring branch forks, context compactions, and terminations, the directory maintains exact reference counts for cached blocks, preventing the eviction of prefixes held by paused agents while promptly recycling discarded history.
- Roofline Iteration-Time Modeling: Rather than relying on coarse queue depths, an online planner profiles separate iteration-time models for cached and uncached tokens. Sizing the prefill-decode split over rolling 10-minute planning windows, the planner converges on optimal node ratios without requiring expensive offline profiling runs.
Empirical Benchmarks Across Coding, Math, and Computer Use
The authors evaluated Evolving Infra on 36 NVIDIA GB300 GPUs (configured across nine 4-GPU nodes using 4-way tensor parallelism) serving GLM-5.2 and Kimi K2.6 models quantized to NVFP4 with FP8 KV caches and multi-token prediction speculative decoding. Benchmark workloads spanned three representative agent domains: coding tasks via Claude Code real-user traces, theorem proving on PutnamBench with mini-SWE-agent, and multimodal UI navigation on OSWorld 2.0. Additional evaluations verified scaling up to 256 physical GPUs and 3,072 simulated GPUs.
Across these environments, the authors reported substantial performance improvements over NVIDIA Dynamo v1.5.1:
- Throughput Gains: Evolving Infra sustained 1.10× to 2.73× the throughput of Dynamo across stationary workloads, achieving up to 1.38× the throughput of optimal static SGLang splits.
- Latency Reduction: Mean end-to-end request latency fell by 1.15× to 4.12× compared to Dynamo, with reductions reaching 3.80× to 4.12× on Claude Code traces and 2.34× to 2.69× on OSWorld tasks.
- Time-to-First-Token: 95th-percentile time-to-first-token (P95 TTFT) improved by up to 13.92× over Dynamo, driven by an 81% to 95% reduction in lost KV reuse.
- Profiling Efficiency: The online planner eliminated the need for offline workload profiling, which the authors calculated would require 10,752 GPU-hours on 256 GPUs for a 3-hour trace.
Implications for University HPC Centers and Research Computing
The findings carry immediate operational significance for academic supercomputing facilities, university high-performance computing (HPC) centers, and institutional AI testbeds. As research teams transition from standard batch training to automated scientific workflows—such as automated literature synthesis, autonomous laboratory robotics control, and generative simulation harnesses—shared campus clusters face unpredictable, bursty inference demand. Institutional administrators evaluating emerging agentic AI benchmarks and research automation must contend with severe hardware underutilization when dedicating fixed GPU partitions to specific models or phases.
Adopting symmetric serving and memory-tiering architectures allows campus facilities to maximize return on expensive accelerator investments. High-capacity host-to-GPU memory tiering offers a practical architectural path for universities constrained by GPU memory budgets, enabling large-context multi-agent workflows without requiring dedicated HBM expansions on every compute node. Furthermore, as research institutions integrate open-source GPU infrastructure and cluster orchestration frameworks, the availability of Evolving Infra as an open-source system (hosted in the MIT Han Lab repository) provides university systems engineers with transparent, reproducible code for local deployment. These capabilities support broader institutional governance efforts within frontier AI safety and governance and help maintain reliable computational pipelines for AI and ML research outputs within academic research data infrastructure.
Sources
- Kang, H., Xu, W., Li, Z., Du, Z., Li, H., Xia, T., Bian, S., Krishna, T., Mao, H., Stoica, I., and Han, S. (2026). Evolving Infra: Agent-Adaptive LLM Serving at Scale. arXiv preprint arXiv:2611.12304v2 [cs.DC]. https://arxiv.org/abs/2611.12304 (Accessed 2026-11-14).
- MIT Han Lab. (2026). Evolving Infra System Implementation. GitHub Repository. https://github.com/mit-han-lab/EvolvingInfra (Accessed 2026-11-14).
- Anthropic. (2024). Computer Use Reference Implementation and OSWorld Benchmarks. https://github.com/anthropics/anthropic-quickstarts (Accessed 2026-11-14).
- Mooncake Project. (2024). A KVCache-Centric Disaggregated Architecture for LLM Serving. https://github.com/kvcache-ai/Mooncake (Accessed 2026-11-14).
