On November 11, 2026, researchers from Argonne National Laboratory, Clemson University, and the University of Louisiana at Lafayette detailed AskALCF, an agentic log-analysis and continuous-improvement framework for supercomputing support. Presented in an arXiv preprint (arXiv:2611.15700v1) accepted at the HUST 2026 workshop at Supercomputing 2026 (SC26), the system analyzes production user interactions, isolates failure causes across retrieval, documentation, prompts, model behavior, and platform tools, and guides governed repairs under human oversight.
Diagnosing Failures Beyond Static Retrieval
Leadership computing facilities like the Argonne Leadership Computing Facility (ALCF) support thousands of scientists across supercomputers whose hardware partitions, programming environments, and software modules evolve rapidly. While earlier tools like AskHPC demonstrated the promise of retrieval-augmented generation (RAG) for user assistance, deployed systems remain largely static. When a language model produces an inadequate answer, aggregate quality scores flag the defect but fail to identify whether the breakdown resulted from missing documentation, poor retrieval ranking, prompt ambiguity, model hallucination, or an infrastructure timeout.
AskALCF couples interactive production Q&A with an automated diagnostic pipeline and ticket re-triage workflow. By converting user traces and support tickets into structured operational evidence, the framework routes each identified defect to the engineering team responsible for that subsystem.
Two-Scope Sufficiency and the 28-Rule Attribution Tree
At the core of AskALCF is an ordered decision tree of 28 diagnostic rules with first-match semantics. Rather than relying on end-to-end evaluations, the pipeline applies a two-scope sufficiency test measuring retrieval at both the production result level (R) and across the entire facility documentation corpus (C). This separation decouples whether information exists in the facility knowledge base from whether the search pipeline succeeded in retrieving it.
The attribution logic enforces a strict hierarchy: platform service and tool errors are evaluated first to prevent infrastructure outages from being misattributed to model hallucinations. Facility-scope and clarification requirements are inspected next, followed by the sufficiency pivot separating retrieval from knowledge gaps. Generative answer quality is assessed only after retrieval is confirmed to be sufficient.
| Attribution Outcome | Decision Criteria and Sufficiency State | Actionable Repair Site |
|---|---|---|
| knowledge | Production result and wider corpus probe both lack documentation (R=False, C=False). | Documentation authoring team writes missing pages. |
| retrieval | Authoritative information exists in the corpus but was not returned (R=False, C=True). | Retrieval engineering fixes query formulation, indexing, or ranking. |
| prompt | Deterministic policy violation occurs ahead of sufficiency pivot (e.g., wrong system assumed). | Prompt policy updates conversation instructions. |
| model | Retrieval was sufficient (R=True) but the model hallucinated non-existent modules or contradicted context. | Model serving adjustments, guardrails, or fine-tuning. |
| tools_mcp | Backend tool execution, schema mismatch, or authentication failure prevented execution. | Platform infrastructure and interface engineering. |
| no_failure | Affirmative verification that retrieval was sufficient and the answer was faithful and accurate. | None (turn concluded successfully). |
| unknown | Evidence is conflicting or insufficient to support a definitive attribution. | Escalated for human expert review. |
Agentic Auditing and Governed Harness Training
To audit the rule tree’s verdicts, AskALCF deploys an agent judge that probes the documentation corpus to surface evidence omitted from initial retrieval results. The authors found that uncalibrated model confidence scores provided poor gating thresholds. Instead, AskALCF restricts the agent judge using a deterministic, rule-conditioned override gate: the judge may only overturn the rule tree on seven specific rule classes whose inputs it can independently verify, boosting agreement by 7 to 9 percentage points without introducing regressions.
System improvements are refined through human-guided ‘harness training,’ which updates the surrounding software harness, rule tree, and prompt policies rather than altering model weights. Reviewers pre-register target turns requiring correction and control turns that must remain unchanged, using null edits as a noise floor across 16 experimental arms.
Evaluated on 2,174 production support turns, the refined harness reduced unattributable outcomes from 41.4% (900 turns) to 14.2% (309 turns). Across a 150-turn expert reference set, agreement rose from 108/150 (72%) to 123/150 (82%), while accuracy on a held-out evaluation set increased from 67% to 77%.
Operational Lessons for Research Computing Facilities
AskALCF’s deployment yields a critical finding for university supercomputing centers and national laboratories: across evaluated queries, 76% of gold standard facts missing from retrieved context were already present within the facility documentation corpus. Only 24% reflected true documentation omissions. Facilities seeking to improve support should therefore prioritize retrieval engineering and indexing over writing redundant documentation.
The framework also unifies conversational support with ticket resolution. In an analysis of 2,211 facility ticket threads, researchers found that 18% of tickets marked complete were not actually resolved, while 319 resolved issues had never been closed. Triaging ticket content through agentic workflows enables automated response drafting with verified citations before expert sign-off, strengthening institutional support for sustainable HPC operations. Such systematic support is crucial as researchers navigate heterogeneous accelerators, deploy Kokkos performance-portability abstractions, and maintain rigorous computational reproducibility across complex workflows.
Sources
- Jiang, J., Bondapalli, A., Rauniyar, K., Keceli, M., Ghadar, Y., Som, H. S., Vishwanath, V., Papka, M. E., Ge, R., and Zheng, H. (2026). AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support. arXiv preprint arXiv:2611.15700v1. https://arxiv.org/abs/2611.15700 (Checked: 2026-11-15).
