On November 12, 2026, University of Illinois Urbana-Champaign computer science researchers Geoffrey Challen and Craig Zilles released a preprint on arXiv titled Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline. Their study introduces a dual-agent architecture that conducts interactive, typed oral examinations at scale without human examiner fatigue, offering academic departments a practical blueprint to test student reasoning rather than static text outputs amid pervasive generative AI.
What happened
Oral examinations remain compelling in higher education because they allow instructors to probe understanding and clarify ambiguous answers in real time. However, oral exams rarely scale due to linear labor demands and human grader inconsistency: faculty face fatigue examining individual students, and evaluators frequently award divergent marks to identical answers.
To eliminate this bottleneck, Challen and Zilles built a chat-based examination pipeline powered by large language models. Students complete live, typed interactive dialogues evaluating their ability to support an argument. The architecture decouples examination into two cooperating agents to protect assessment integrity:
- Interviewer agent: Interacts directly with the student via chat. Lacking access to rubrics or reference answers, it cannot leak answers or hints, receiving only conversational guidance on what to ask next.
- Evaluator agent: Operates behind the scenes with full access to assignment criteria, reference answers, and specifications-grading rubrics (Excellent, Satisfactory, Needs Improvement, Unsatisfactory). It tracks progress turn by turn, manages pacing, decides when to probe or advance, and directs the interviewer.
Methodology and key findings
Before student deployment, the authors tested the architecture using eleven simulated LLM student personas across five legitimate capability tiers and six adversarial archetypes (prompt injectors, answer extractors, and fabricators). Crucially, testing required a fictional assessment topic; otherwise, LLM personas drew upon pre-trained knowledge rather than prompt constraints. System prompts were refined until all legitimate personas achieved target grades and adversarial personas were scored Unsatisfactory.
The system was deployed during Spring 2026 in an AI literacy seminar for non-majors (N = 11) at the University of Illinois Urbana-Champaign across three formative assignments and one proctored summative exam. Students finished 41 evaluated attempts covering conceptual topics including the Turing test and machine creativity. Telemetry showed a median exam required 20 LLM calls (80,000 tokens) and lasted 19 to 30 minutes, costing a median of $0.27 per attempt via GPT-5.2 on Azure OpenAI Service. Deployed outcomes across 40 graded attempts were 20 Excellent, 10 Satisfactory, 5 Needs Improvement, and 5 Unsatisfactory marks.
A post-semester audit exposed an essential nuance. When forty transcripts were re-graded by an independent retrospective LLM judge (Claude Fable 5), exact agreement with deployed grades was only 35%, shifting eighteen top marks to Satisfactory due to strict scaffolding rules. Repeating the audit with retrospective GPT-5.2 confirmed the divergence stemmed from evaluation timing rather than model architecture: live turn-by-turn grading suffered from leniency because probing terse students inadvertently supplied conversational scaffolding. Blind human re-grading by both authors on ten transcripts showed 60% exact agreement, paralleling cross-model AI variance. The authors recommend using live evaluators strictly for conversation management and deriving the official grade of record from a retrospective, whole-transcript second read.
Why it matters for research practice
As generative tools make unproctored written homework difficult to verify, research institutions and teaching faculties are actively rethinking assessment architecture. Relying on commercial AI detectors for research integrity has repeatedly fallen short due to false-positive risks and stylistic bias. Chat-based conversational exams paired with proctored computer labs offer a robust alternative, testing student comprehension through dynamic defense rather than static homework submissions.
The study also demonstrates the necessity of disciplined AI output verification before rolling out automated tools in academic environments. The finding that conversational evaluators inadvertently coach struggling students underscores the fragility of single-agent grading pipelines. Establishing rigorous post-hoc auditing protocols ensures that automated assessment remains transparent, aligned with grading rubrics, and safeguarded against conversational drift.
Finally, these findings have direct implications for faculty governance and responsible conduct of research training across higher education. Because the two-agent architecture operates on plain-text rubrics rather than discipline-specific code execution, the framework can be adapted across social sciences, humanities, and laboratory disciplines. Implementing independent second-look grading protocols provides a defensible mechanism to protect academic standards while keeping per-student assessment costs under thirty cents.
Sources
- Challen, G., and Zilles, C. (2026). Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline. arXiv preprint arXiv:2611.17220. https://arxiv.org/abs/2611.17220
