At Supercomputing 2026 (SC26) in Chicago on November 16, 2026, AI data architecture firm WEKA, independent analyst group NAND Research, and semiconductor research firm SemiAnalysis unveiled the Inference Economics Scorecard and the open-source AgentX benchmark. Designed to replace antiquated GPU utilization metrics, the collaborative framework establishes an evaluation methodology that measures production AI infrastructure by useful work delivered, prefix-cache retention, and energy efficiency rather than raw chip activity.
New Benchmarking Framework Measures Useful Output Over Raw GPU Churn
As enterprise and academic computing centers shift budgets from training to multi-turn agentic workloads, conventional monitoring tools have created an operational measurement trap. Standard metrics such as Model FLOPS Utilization (MFU) and raw GPU occupancy track whether silicon is active, but fail to discern whether that activity produces fresh tokens or merely recomputes discarded context. To address this blind spot, WEKA and NAND Research introduced the Inference Economics Scorecard on November 16, 2026, defining nine standardized operational indicators alongside four strict testing disciplines to evaluate systems on finished output.
Alongside the scorecard, SemiAnalysis released its open-source AgentX benchmarking harness within the InferenceX platform. Designed to simulate multi-turn, multi-agent workflows such as complex programming and literature synthesis pipelines, AgentX replays authentic traces using WEKA’s open kvlab format without exposing proprietary code or sensitive prompts. Executed against complete serving stacks across multi-hour profiles, the combined framework gives computing centers reproducible criteria for sizing memory hierarchies, storage tiers, and accelerator clusters.
Key Technical Architecture: Nine Metrics, Four Rules, and AgentX Replay DAGs
The Inference Economics Scorecard defines nine interconnected metrics reflecting real data center constraints: tokens per watt, tokens per rack, cost per inference, GPU goodput, time to first token (TTFT) and end-to-end latency, memory efficiency, operational density, agent concurrency, and context retention. The framework enforces four mandatory testing constraints across all evaluations:
- Identical Workloads and Service Targets: Competing systems must process the same request distributions, concurrency levels, and latency ceilings.
- Full Serving-Stack Boundary: Measurements must encompass the entire architecture, including accelerators, host memory, networking fabric, storage tiers, and cooling overhead.
- Mandatory Eight-Hour Endurance Runs: Benchmarks must run continuously for at least eight hours to expose thermal degradation, memory pressure, and timeout cascades.
- Exclusively Successful Output: Only completed responses meeting quality thresholds and latency targets count toward throughput; failed calls, timeouts, and dropped requests are logged as system losses.
To feed realistic workloads into this framework, SemiAnalysis developed the AgentX harness. Unlike synthetic benchmarks that stream uniform sequences, AgentX models complex agent structures captured through opt-in proxies. The v1.0 dataset incorporates 393 real-world Claude Code sessions spanning 135,282 requests. The median request carries 142,016 input tokens and 444 output tokens, with subagents active in 44% of sessions across 1,697 execution groups.
During replay, NVIDIA AIPerf converts session traces into directed acyclic graphs (DAGs). Main-agent requests form sequential chains, while parallel subagents branch outward and synchronize at join gates before dependent steps proceed. Crucially, the capture pipeline hashes conversation contexts into 64-token session-scoped blocks, enabling prefix-cache matching without storing raw text, aligning with strict standards for AI data provenance and research governance.
Eight-Hour HGX B300 Soak Test Demonstrates KV-Cache Retention Gains
To validate the scorecard in production conditions, cloud provider Nebius joined WEKA to conduct an eight-hour continuous soak test on an eight-GPU NVIDIA HGX B300 node serving DeepSeek-V4-Pro. The benchmark simulated 512 active client sessions with a scheduler limit of 32 simultaneous requests, comparing conventional high-bandwidth memory (HBM-only) serving against WEKA NeuralMesh with Augmented Memory Grid, a distributed NVMe-backed key-value (KV) cache tier.
Under the HBM-only configuration, context eviction forced GPUs to recompute 60.0% of prefill cycles from history the cluster had already processed earlier. Offloading warm KV-cache blocks across the NVMe cluster reduced prefill recomputation to 6.8%, driving a 3.7-fold increase in output tokens per GPU-hour.
| Evaluation Dimension | HBM-Only Baseline | WEKA NeuralMesh (Augmented Memory Grid) | Measured Impact |
|---|---|---|---|
| Output Tokens / GPU-Hour | 1.0× (Baseline) | 3.7× Throughput | 270% increase in productive token delivery |
| Requests Meeting TTFT ≤ 5s | 44.0% | 97.0% | Sustained interactive responsiveness for agents |
| Prefill Recomputation Share | 60.0% | 6.8% | 53.2 percentage point drop in redundant compute |
| Time to Produce 8h Token Output | ~30 Hours | 8 Hours | 73.3% reduction in cluster execution time |
| Latency Profile Stability | Degraded as timeouts climbed | Flat TTFT across 8-hour window | Elimination of long-context eviction cascades |
Industry Context: The Transition From Training Era to Inference Economy
The scorecard reflects a macro-level realignment in AI infrastructure. While foundation model training previously dominated capital budgets, research from 451 Research by S&P Global indicates inference will expand from 34% of enterprise AI infrastructure expenditure in 2026 to 46% by 2030, with 41% of organizations procuring dedicated architectures for serving workloads.
This shift introduces severe physical bottlenecks. Accelerator expansion has collided with regional power grids, cooling limits, and extended lead times for high-voltage transformers. Because agentic workflows ingest expanding histories on every turn, GPU memory saturates quickly. When clusters cannot sustain warm KV caches, they waste electricity reconstructing old state, creating efficiency losses that facilities can monitor through computational carbon accounting tools.
Stakeholder Reactions Highlight Shift to Finished Work Metrics
Industry leaders across storage, silicon, and analysis emphasized that transparent output metrics are vital to evaluating AI sustainability. Liran Zvibel, co-founder and CEO of WEKA, noted at SC26 that grading agentic inference with training-era utilization rewards wasted cycles, arguing that useful work per watt and per dollar must become the definitive baseline.
NAND Research founder and principal analyst Steve McDowell stressed that buying infrastructure for peak throughput often results in organizations paying repeatedly for duplicated prefill work. McDowell stated that the scorecard establishes uniform verification criteria across vendors by penalizing failed or timed-out requests over sustained operational windows.
SemiAnalysis founder and CEO Dylan Patel highlighted the necessity of trace-driven testing, noting that synthetic generators cannot replicate the bursty fan-outs, pauses, and deep subagent trees of production systems. Patel confirmed that AgentX will publish ongoing comparative evaluations across major accelerator architectures using the open kvlab standard.
Nebius vice president of product Narek Tatevosyan pointed out that capacity planning must reflect sustained daily output rather than brief benchmark peaks. NVIDIA vice president of storage technology Jason Hardy added that as inference workloads define the AI factory, extending context tiers beyond high-bandwidth memory into distributed storage fabrics will determine infrastructure viability.
Implications for University Research Computing and Institutional IT
For higher education institutions, academic medical centers, and national laboratory facilities, the launch of the Inference Economics Scorecard and AgentX provides critical leverage for procurement and resource allocation. Research computing directors often balance tight budgets between dense GPU nodes and memory fabric investments. The scorecard demonstrates that pairing accelerators with high-speed NVMe cache tiers can deliver higher effective throughput than purchasing additional unaugmented compute nodes.
Furthermore, academic computing facilities supporting autonomous scientific agents—such as literature synthesis, molecular modeling, and bioinformatics coding assistants—can use AgentX traces to establish realistic capacity limits. By measuring agent concurrency and context retention under strict service-level targets, university administrators can prevent system thrashing and allocate compute quotas based on productive output rather than wall-clock GPU occupancy, strengthening institutional data stewardship and storage governance.
The open-source release of kvlab traces and AIPerf tools also elevates experimental rigor. Institutional researchers evaluating caching algorithms, model serving engines, or accelerators can reference standardized traces, aligning with academic standards for computational data availability and reproducibility. As universities align computing operations with regional governance standards like the EU AI Act transparency and risk management obligations, rigorous benchmarking of energy consumption per useful token offers an auditable trail for environmental and financial accounting.
Next Steps and Community Availability
WEKA is demonstrating the Inference Economics Scorecard and kvlab tooling throughout SC26 in Chicago from November 16 through November 20, 2026, at booth #2519. The open-source kvlab repository, including trace capture proxies, anonymization pipelines, and AIPerf loaders, is publicly accessible on GitHub. SemiAnalysis plans to expand AgentX comparative runs across additional accelerator platforms—including AMD Instinct, NVIDIA Blackwell, and custom cloud silicon—with public leaderboards updating continuously on the InferenceX platform through early 2027.
Sources
- PR Newswire: New Research Challenges GPU Utilization as Measure of Inference Infrastructure, checked 2026-11-16
- Unite.AI: The Inference Economics Scorecard Challenges GPU Utilization as an AI Performance Metric, checked 2026-11-16
- SemiAnalysis: AgentX Methodology and Datasets, checked 2026-11-16
- Nebius: Nebius and WEKA Benchmark Shared KV Cache for Agentic Inference on NVIDIA HGX B300, checked 2026-11-16
- WEKA: Agents Don’t Benchmark Like Chatbots: Meet kvlab, checked 2026-11-16







