Skip to main content
v2026.11,986 entries · CC-BY 4.0

Editorial · CASRAI · AI and ML research outputs

Preprint Evaluates Host-Tier KV Restoration for Multi-Model Agent Workflows on NPUs

A November 2026 preprint tests host-tier KV cache restoration on Rebellions NPUs, finding major gains during overload alongside tail latency trade-offs.

Published 16 Nov 2026· 6 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

On November 11, 2026, researchers Sungwon Woo and Seyeon Kim released an empirical preprint evaluating host-tier key-value (KV) cache restoration for multi-model AI agent workflows on static-graph neural processing units (NPUs). Conducted on Rebellions ATOM+ accelerators, the study demonstrates that offloading evicted KV caches to host DRAM rescues server queues from collapse during overload, while creating distinct tail latency trade-offs for branching agent pipelines.

Empirical Evaluation of Host-Tier KV Offloading on NPUs

In multi-agent systems, workflows run as directed acyclic graphs (DAGs) where specialized language models handle planning, code execution, and response synthesis. Because adjacent execution steps share extensive context prefixes such as system prompts, instructions, and tool outputs, serving efficiency depends on reusing the key-value (KV) cache of those prefixes. While dynamic GPU runtimes allocate cache dynamically, static-graph NPUs compile execution graphs ahead of time with rigid on-device KV block limits.

When device memory saturates under multi-model traffic, static-graph serving stacks evict older prefixes, forcing subsequent dependent calls to recompute prefill from scratch. In their preprint titled ‘Recompute or Restore? Host-Tier KV for Multi-Model Agent Workflows on a Static-Graph NPU Serving Stack’, published on the arXiv preprint server, Woo and Kim investigate whether restoring evicted KV blocks from host system memory over PCIe outperforms recomputing prefill on accelerator hardware.

The authors found that in baseline serving without host caching, repeated prefill dominates accelerator workloads under heavy traffic. In an overload experiment evaluating a fork workflow, 89.5% of all executed prompt tokens processed by the planner model had already been materialized in earlier stages. Prefill calls composed purely of repeated tokens consumed 59.3% of planner API time, triggering severe queue congestion and request timeouts.

Serving Architecture and Hardware Testbed

The evaluation was conducted on a cluster of Rebellions ATOM+ (RBLN-CA22) PCIe accelerator cards running kernel driver KMD 3.0.0 and vllm-rbln version 0.10.2.post2. The multi-model topology paired an 8-billion parameter planner model (Alibaba Qwen3-8B in bfloat16) partitioned across two ATOM+ cards via tensor parallelism (TP2) with a 0.6-billion parameter executor model (Qwen3-0.6B) on a third card.

To establish host-tier caching without altering underlying firmware, the researchers built a synchronous host-DRAM tier interfaced through the runtime’s public key-value copy hooks. The architecture bridges device memory management with eviction and block-pinning protocols adapted from vLLM, exchanging prefixes across host DRAM and accelerator memory in 128-token block allocations.

The study evaluated four synthetic workflow DAGs: linear chain pipelines, parallel fork structures, mixed workloads, and bursty arrival distributions. Benchmarking was performed using a synthetic benchmark replaying deterministic workflow traces across multiple random seeds, ensuring strict computational reproducibility across varied queue depths.

Overload Recovery and Tail Latency Penalties

The host DRAM tier delivered substantial stability gains during overload. Under stock serving (B0), offering 0.55 fork workflows per second caused server throughput to collapse, achieving only 22.8% service-level objective (SLO) attainment with 536 errors out of 1,200 arrivals. Adding the host tier restored SLO attainment to 100.0% with zero failed requests. In a mixed-burst overload test at 0.65 workflows per second, baseline SLO attainment fell to 9.4% with 291 errors; enabling the host tier raised SLO attainment to 99.8% with only two dropped requests.

Across overload conditions, total planner prefill API duration dropped from 1,115–1,708 seconds in the baseline system down to 262–265 seconds with host restoration. However, sustainable common-load traffic revealed an important trade-off between median latency and tail latency.

While host prefix restoration reduced median (P50) latency by approximately 20% across chain and fork workloads, it increased 99th-percentile (P99) tail latency by 15.6% to 17.5% in fork workflows. Because synchronous transfers execute on the same engine thread that handles prefill and decoding, transferring cached blocks prevents the thread from advancing decodes for concurrent requests. In addition, board-level power measurements showed that the host tier lowered gross energy consumption per offered workflow by 3.7% to 10.5% compared to stock serving, as detailed below.

Workload Condition Serving Policy SLO Attainment (%) Errors P50 Latency Delta P99 Latency Delta Energy Change
KV64 Fork Overload (0.55 wf/s) Stock Serving (B0) 22.8% 536 Baseline Baseline Baseline
KV64 Fork Overload (0.55 wf/s) Host-Tier DRAM (F6+host) 100.0% 0 Stable (< 10s) Stable (< 15s) Reduced
KV32 Burst Overload (0.65 wf/s) Stock Serving (B0) 9.4% 291 Baseline Baseline Baseline
KV32 Burst Overload (0.65 wf/s) Host-Tier DRAM (F6+host) 99.8% 2 Stable (< 8s) Stable (< 12s) Reduced
Common Load Chain (0.72 wf/s) F6+host vs F6 Base 100.0% 0 −19.8% +5.3% avg −3.8% J/wf
Common Load Fork (0.36 wf/s) F6+host vs F6 Base 100.0% 0 −19.8% +16.6% avg −6.7% J/wf
Common Load Burst (0.44 wf/s) F6+host vs F6 Base 100.0% 0 −26.5% −38.5% −10.5% J/wf

System Context and Predictive Modeling Limits

Managing memory hierarchies for large language model inference is an active research focus. Frameworks such as SGLang HiCache and InferCept explore hierarchical caching across host memory and NVMe to mitigate GPU memory bottlenecks. In parallel, multi-agent frameworks like KVFlow optimize prefix retention across collaborative workflows.

The Woo and Kim study highlights non-GPU static-graph accelerators, which are gaining interest in cost-conscious research clusters. Commercial NPU architectures, such as the Rebellions vLLM stack, offer high energy efficiency for pre-compiled static kernels, but lack the runtime agility of dynamic GPU memory allocation.

To test whether system performance could be predicted analytically, the authors built a discrete-event simulation model, freezing predictions prior to hardware runs. The model forecasted P50 and P99 latencies within a 15% error tolerance across 24 common-load tests. However, it misordered three policy comparisons and overpredicted the benefit of a packaging optimization designed to cut store overheads (predicting an 11% latency reduction versus the 3.1% reduction measured), demonstrating that engine thread contention defies simple algebraic estimates.

What It Means for Research Offices and Academic Computing

These findings provide clear guidance for research computing facilities and university research offices planning AI infrastructure:

  • Hardware Diversification: Amid high GPU procurement costs and power constraints, static-graph NPUs offer viable alternatives. The study proves that host-memory KV tiering enables commercial NPUs to handle multi-model agent workloads effectively without requiring costly on-chip high-bandwidth memory expansions.
  • Research Environment Configuration: Systems engineers maintaining an institutional computational environment must configure serving stacks around workload topology. Workflows with parallel forks require asynchronous KV transfer paths to avoid degrading tail latency for multi-user clusters.
  • Rigorous AI Experimentation: The authors’ methodology of freezing simulation models before execution and testing across multiple seeds establishes a strong protocol for any reproducible AI experiment in systems research.
  • Grant Compute Budgeting: Research offices budgeting compute grants for multi-agent workflows should confirm whether host-tier caching is supported. Enabling host caching can lower board energy use by up to 10.5% and prevent queue collapse during batch arrivals.

Next Steps in Accelerator Serving Middleware

Following the preprint release, research attention focuses on integrating asynchronous memory pipelines into production NPU runtimes. Newer vendor implementations, including vllm-rbln 0.12 with LMCache connectors, aim to overlap PCIe transfers with compute operations using dedicated direct memory access (DMA) engines. Future studies will test whether decoupling KV transfers from the primary engine thread eliminates fork tail latency spikes while sustaining high throughput.

Sources

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →