On November 11, 2026, researchers Sungwon Woo and Seyeon Kim released an empirical preprint evaluating host-tier key-value (KV) cache restoration for multi-model AI agent workflows on static-graph neural processing units (NPUs). Conducted on Rebellions ATOM+ accelerators, the study demonstrates that offloading evicted KV caches to host DRAM rescues server queues from collapse during overload, while creating distinct tail latency trade-offs for branching agent pipelines.
Empirical Evaluation of Host-Tier KV Offloading on NPUs
In multi-agent systems, workflows run as directed acyclic graphs (DAGs) where specialized language models handle planning, code execution, and response synthesis. Because adjacent execution steps share extensive context prefixes such as system prompts, instructions, and tool outputs, serving efficiency depends on reusing the key-value (KV) cache of those prefixes. While dynamic GPU runtimes allocate cache dynamically, static-graph NPUs compile execution graphs ahead of time with rigid on-device KV block limits.
When device memory saturates under multi-model traffic, static-graph serving stacks evict older prefixes, forcing subsequent dependent calls to recompute prefill from scratch. In their preprint titled ‘Recompute or Restore? Host-Tier KV for Multi-Model Agent Workflows on a Static-Graph NPU Serving Stack’, published on the arXiv preprint server, Woo and Kim investigate whether restoring evicted KV blocks from host system memory over PCIe outperforms recomputing prefill on accelerator hardware.
The authors found that in baseline serving without host caching, repeated prefill dominates accelerator workloads under heavy traffic. In an overload experiment evaluating a fork workflow, 89.5% of all executed prompt tokens processed by the planner model had already been materialized in earlier stages. Prefill calls composed purely of repeated tokens consumed 59.3% of planner API time, triggering severe queue congestion and request timeouts.
Serving Architecture and Hardware Testbed
The evaluation was conducted on a cluster of Rebellions ATOM+ (RBLN-CA22) PCIe accelerator cards running kernel driver KMD 3.0.0 and vllm-rbln version 0.10.2.post2. The multi-model topology paired an 8-billion parameter planner model (Alibaba Qwen3-8B in bfloat16) partitioned across two ATOM+ cards via tensor parallelism (TP2) with a 0.6-billion parameter executor model (Qwen3-0.6B) on a third card.
To establish host-tier caching without altering underlying firmware, the researchers built a synchronous host-DRAM tier interfaced through the runtime’s public key-value copy hooks. The architecture bridges device memory management with eviction and block-pinning protocols adapted from vLLM, exchanging prefixes across host DRAM and accelerator memory in 128-token block allocations.
The study evaluated four synthetic workflow DAGs: linear chain pipelines, parallel fork structures, mixed workloads, and bursty arrival distributions. Benchmarking was performed using a synthetic benchmark replaying deterministic workflow traces across multiple random seeds, ensuring strict computational reproducibility across varied queue depths.
Overload Recovery and Tail Latency Penalties
The host DRAM tier delivered substantial stability gains during overload. Under stock serving (B0), offering 0.55 fork workflows per second caused server throughput to collapse, achieving only 22.8% service-level objective (SLO) attainment with 536 errors out of 1,200 arrivals. Adding the host tier restored SLO attainment to 100.0% with zero failed requests. In a mixed-burst overload test at 0.65 workflows per second, baseline SLO attainment fell to 9.4% with 291 errors; enabling the host tier raised SLO attainment to 99.8% with only two dropped requests.
Across overload conditions, total planner prefill API duration dropped from 1,115–1,708 seconds in the baseline system down to 262–265 seconds with host restoration. However, sustainable common-load traffic revealed an important trade-off between median latency and tail latency.
While host prefix restoration reduced median (P50) latency by approximately 20% across chain and fork workloads, it increased 99th-percentile (P99) tail latency by 15.6% to 17.5% in fork workflows. Because synchronous transfers execute on the same engine thread that handles prefill and decoding, transferring cached blocks prevents the thread from advancing decodes for concurrent requests. In addition, board-level power measurements showed that the host tier lowered gross energy consumption per offered workflow by 3.7% to 10.5% compared to stock serving, as detailed below.
| Workload Condition | Serving Policy | SLO Attainment (%) | Errors | P50 Latency Delta | P99 Latency Delta | Energy Change |
|---|---|---|---|---|---|---|
| KV64 Fork Overload (0.55 wf/s) | Stock Serving (B0) | 22.8% | 536 | Baseline | Baseline | Baseline |
| KV64 Fork Overload (0.55 wf/s) | Host-Tier DRAM (F6+host) | 100.0% | 0 | Stable (< 10s) | Stable (< 15s) | Reduced |
| KV32 Burst Overload (0.65 wf/s) | Stock Serving (B0) | 9.4% | 291 | Baseline | Baseline | Baseline |
| KV32 Burst Overload (0.65 wf/s) | Host-Tier DRAM (F6+host) | 99.8% | 2 | Stable (< 8s) | Stable (< 12s) | Reduced |
| Common Load Chain (0.72 wf/s) | F6+host vs F6 Base | 100.0% | 0 | −19.8% | +5.3% avg | −3.8% J/wf |
| Common Load Fork (0.36 wf/s) | F6+host vs F6 Base | 100.0% | 0 | −19.8% | +16.6% avg | −6.7% J/wf |
| Common Load Burst (0.44 wf/s) | F6+host vs F6 Base | 100.0% | 0 | −26.5% | −38.5% | −10.5% J/wf |
System Context and Predictive Modeling Limits
Managing memory hierarchies for large language model inference is an active research focus. Frameworks such as SGLang HiCache and InferCept explore hierarchical caching across host memory and NVMe to mitigate GPU memory bottlenecks. In parallel, multi-agent frameworks like KVFlow optimize prefix retention across collaborative workflows.
The Woo and Kim study highlights non-GPU static-graph accelerators, which are gaining interest in cost-conscious research clusters. Commercial NPU architectures, such as the Rebellions vLLM stack, offer high energy efficiency for pre-compiled static kernels, but lack the runtime agility of dynamic GPU memory allocation.
To test whether system performance could be predicted analytically, the authors built a discrete-event simulation model, freezing predictions prior to hardware runs. The model forecasted P50 and P99 latencies within a 15% error tolerance across 24 common-load tests. However, it misordered three policy comparisons and overpredicted the benefit of a packaging optimization designed to cut store overheads (predicting an 11% latency reduction versus the 3.1% reduction measured), demonstrating that engine thread contention defies simple algebraic estimates.
What It Means for Research Offices and Academic Computing
These findings provide clear guidance for research computing facilities and university research offices planning AI infrastructure:
- Hardware Diversification: Amid high GPU procurement costs and power constraints, static-graph NPUs offer viable alternatives. The study proves that host-memory KV tiering enables commercial NPUs to handle multi-model agent workloads effectively without requiring costly on-chip high-bandwidth memory expansions.
- Research Environment Configuration: Systems engineers maintaining an institutional computational environment must configure serving stacks around workload topology. Workflows with parallel forks require asynchronous KV transfer paths to avoid degrading tail latency for multi-user clusters.
- Rigorous AI Experimentation: The authors’ methodology of freezing simulation models before execution and testing across multiple seeds establishes a strong protocol for any reproducible AI experiment in systems research.
- Grant Compute Budgeting: Research offices budgeting compute grants for multi-agent workflows should confirm whether host-tier caching is supported. Enabling host caching can lower board energy use by up to 10.5% and prevent queue collapse during batch arrivals.
Next Steps in Accelerator Serving Middleware
Following the preprint release, research attention focuses on integrating asynchronous memory pipelines into production NPU runtimes. Newer vendor implementations, including vllm-rbln 0.12 with LMCache connectors, aim to overlap PCIe transfers with compute operations using dedicated direct memory access (DMA) engines. Future studies will test whether decoupling KV transfers from the primary engine thread eliminates fork tail latency spikes while sustaining high throughput.
Sources
- Woo, S., and Kim, S. (2026). Recompute or Restore? Host-Tier KV for Multi-Model Agent Workflows on a Static-Graph NPU Serving Stack. arXiv:2611.15126. https://arxiv.org/abs/2611.15126 (Accessed 2026-11-16).
- Rebellions. (2026). KV Cache Offloading (vLLM Support Documentation). Rebellions AI Docs. https://docs.rbln.ai/latest/software/model_serving/vllm_support/features/kv-offloading.html (Accessed 2026-11-16).
- vLLM Project. (2025). KV Offloading Connector and LRU Offloading Manager. GitHub Repository. https://github.com/vllm-project/vllm (Accessed 2026-11-16).
- Xie, Z. (2025). SGLang HiCache: Fast Hierarchical KV Caching With Your Favorite Storage Backends. LMSYS Org. https://lmsys.org/blog/2025-09-10-sglang-hicache/ (Accessed 2026-11-16).
- Abhyankar, R., He, Z., Srivatsa, V., Zhang, H., and Zhang, Y. (2024). InferCept: Efficient Intercept Support for Augmented Large Language Model Inference. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), PMLR 235. https://proceedings.mlr.press/v235/abhyankar24a.html (Accessed 2026-11-16).
- Yang, A., et al. (2025). Qwen3 Technical Report. arXiv:2505.09388. https://arxiv.org/abs/2505.09388 (Accessed 2026-11-16).







