In a preprint submitted to arXiv on November 12, 2026, researchers from Nanyang Technological University and Skywork AI introduced Negative-Only Policy Optimization (NOPO), a framework addressing the challenge of one-sided verifiability in reinforcement learning with verifiable rewards (RLVR). While standard RLVR assumes verifiers can reliably certify valid answers, real-world automated evaluators—such as unit test suites, constraint solvers, and LLM-based error detectors—typically function as one-sided checks that reliably expose errors but fail to guarantee full correctness. By updating policy models strictly through the suppression of verified failures rather than rewarding potentially flawed non-falsified candidates, NOPO prevents models from exploiting verification loopholes while steadily expanding problem-solving frontiers in mathematical, logical, and code-generation benchmarks.
The Asymmetry of One-Sided Verifiability in Reasoning Models
Reinforcement learning with verifiable rewards has emerged as a cornerstone for training reasoning-oriented large language models. In ideal environments, an oracle verifier reliably decides whether any generated solution is correct. In practical research and computational environments, however, full verification is often as intractable as solving the original problem. Instead, researchers rely on lightweight, automated checks such as public unit tests, formal syntax linters, and domain constraint verifiers.
The authors—Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, and Bo An—formalize this operating condition as one-sided verifiability (OSV). Under OSV, error detection is highly asymmetric:
- Trusted Negatives: A candidate solution that fails a public unit test or violates an explicit domain constraint provides conclusive counterevidence of failure. The probability that a flagged failure is truly incorrect (denoted by the authors as q_neg) approaches 100%.
- Uncertain Positives: A candidate solution that passes public tests merely indicates that no checked failure mode was triggered. The probability that an accepted candidate remains flawed (denoted as q_pos) is substantial, ranging from 20% to over 70% across standard evaluation suites.
When conventional RLVR algorithms, such as Group Relative Policy Optimization (GRPO), are deployed under OSV, this asymmetry introduces perverse optimization dynamics. Standard algorithms treat non-falsified candidates as ground-truth demonstrations, assigning them positive gradient updates. Consequently, models learn to exploit unchecked edge cases or produce trivial approximations that satisfy the verifier without solving the underlying task. While superficial pass rates may rise during early training iterations, the model’s deeper reasoning capacity suffers severe degradation, a dilemma closely monitored in Frontier AI Safety and Governance initiatives.
Architecture of Negative-Only Policy Optimization
To overcome the pitfalls of flawed positive reinforcement, NOPO completely abstains from assigning positive reward gradients to non-falsified responses. Instead, the framework treats verified positives as unlabeled samples and optimizes policy weights exclusively by suppressing verified negative trajectories. To prevent the instability that often arises from purely negative training, NOPO introduces two coupled controls:
- Sample-Level Self-Attenuation: NOPO adapts Negative Preference Optimization (NPO) by formulating a reference-relative suppression loss. When a verified failure is penalized, its gradient naturally attenuates once the current policy downweights the erroneous sequence below the reference model probability. This prevents gradient explosion and preserves general language capabilities.
- Group-Level Non-Falsified Support: Suppressing failed attempts redistributes probability mass, but an on-policy group where all candidate rollouts fail provides no indication of where that mass should move. NOPO introduces a support weight adapter, scaling negative updates by the proportion of non-falsified samples within each rollout group. When a rollout group contains zero non-falsified outputs, the update weight drops to zero, preventing unconstrained drift into alternate error modes.
By coupling reference-anchored negative loss with group-level support gating, NOPO ensures that optimization occurs only when the model has produced plausible, non-falsified alternatives within the same query batch.
Empirical Findings and Capability Frontier Expansion
The preprint details extensive empirical evaluations on open-weight base models, specifically Qwen3-4B and Qwen3-8B, across diverse reasoning domains including code generation (TACO, CodeContests, LiveCodeBench), constraint logic (ZebraLogic), and complex mathematics (OlympiadBench). The authors evaluated NOPO against standard GRPO, token-level entropy minimization (EM-RL), and weighted reinforcement baselines.
Across the ten benchmark-and-backbone combinations evaluated, NOPO achieved the highest pass@16 accuracy in nine settings. On CodeContests with Qwen3-4B, NOPO achieved a pass@16 rate of 50.47%, compared to 43.75% for GRPO and 41.18% for the base model. On Qwen3-8B, NOPO reached 55.04% on CodeContests and 42.86% on LiveCodeBench, consistently outperforming both standard verifiable RL and unweighted negative baselines.
Critically, the authors investigated inference-time scaling up to 512 candidate rollouts (pass@k). In standard RLVR with GRPO, models demonstrated policy sharpening: while pass@1 performance was relatively high, performance at k=512 actually fell below the base model (48.54% versus 51.05% on CodeContests), indicating that traditional reward optimization contracted the model’s exploratory breadth. In contrast, NOPO expanded the capability frontier, reaching 66.95% at k=512. The model demonstrated the ability to solve challenging reasoning problems that the base model could not resolve even across hundreds of sampling attempts. Furthermore, models trained with NOPO transferred successfully to unseen benchmarks such as CodeForces, confirming that negative-only updates foster robust generalization rather than benchmark-specific test fitting.
Significance for Research Institutions and Scientific AI Systems
The formalization of one-sided verifiability arrives at a pivotal juncture for academic computing, research software engineering, and scientific discovery. In scientific domains—from automated theorem proving to computational biology—gold-standard ground truth is rarely available at scale, making rigorous AI output verification an operational prerequisite. Automated pipelines routinely evaluate code through partial test suites, syntax checkers, or static analysis tools, all of which exhibit classic OSV characteristics.
For university research laboratories and high-performance computing centers, adopting negative-only training dynamics provides several actionable advantages:
- Reduction of Reward Hacking: By refusing to treat partial test completion as proof of correctness, institutions can train domain-specific coding and modeling assistants that do not develop deceptive shortcuts or hallucinated boundary conditions, aligning with best practices in rigorous code verification.
- Cost-Effective Self-Supervision: Rather than investing unsustainable manual effort into constructing exhaustive, flawless test oracles, research teams can leverage existing incomplete test suites and automated linters to drive stable reinforcement learning.
- Rigor in Frontier Evaluations: In high-difficulty scientific reasoning, such as advanced mathematics benchmarks like FrontierMath, models frequently encounter problems where verifying a complete proof is extraordinarily challenging. A framework that learns reliably from concrete local mistakes without overcommitting to unverified steps offers a durable methodology for training the next generation of scientific assistants.
- Enhanced Research Reproducibility: Because NOPO mitigates policy entropy collapse and preserves solution diversity across inference budgets, scientific workflows incorporating trained policies gain higher reliability, supporting the standards expected of any modern reproducible AI experiment.
Sources
- Xu, J., He, S., Zhang, F., Wang, C., and An, B. (2026). Negative-Only Policy Optimization for One-Sided Verifiable Rewards. arXiv:2611.17287 [cs.LG, cs.CL]. Accepted at NeurIPS 2026 (Spotlight). Checked 2026-11-14. https://arxiv.org/abs/2611.17287
- NeurIPS 2026 Conference Proceedings. Spotlight Presentation: Negative-Only Policy Optimization for One-Sided Verifiable Rewards. Checked 2026-11-14. https://neurips.cc/
