Written and maintained by CASRAI Editorial Board
Last updated
What the paper is
“Concrete Problems in AI Safety” is a 2016 research paper by Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. It was posted to arXiv in June 2016 (arXiv:1606.06565) under the categories Artificial Intelligence (cs.AI) and Machine Learning (cs.LG). The paper does not attempt to address long-range or speculative risks from advanced AI. Instead, its stated aim is to identify practical research problems that arise from accidents in machine learning systems — cases where a system’s own objective or design causes unintended and harmful behaviour, rather than cases involving malicious use.
The paper organises its five problems by where in a system they originate: a wrong objective function, an objective that is too expensive to evaluate often, or undesirable behaviour during the learning process itself.
The five concrete problems
As stated in the paper’s abstract, the five problems are:
- Avoiding side effects. How to ensure an AI system does not disturb its environment in harmful ways while pursuing its objective, when those effects are not part of what the objective explicitly penalises.
- Avoiding reward hacking. How to ensure a system does not game its own reward function — finding ways to score highly on the specified objective that do not match what its designers actually intended.
- Scalable supervision. How to make efficient use of limited human oversight when the true objective is expensive or impractical to evaluate every time the system acts.
- Safe exploration. How to ensure a system explores its environment, as reinforcement learning systems must, without taking actions that are harmful or irreversible.
- Distributional shift. How to ensure a system recognises and behaves cautiously in situations that differ from its training environment, rather than acting confidently on assumptions that no longer hold.
The paper’s own framing groups the first two under objectives that are wrong, the third under objectives that are too expensive to evaluate frequently, and the last two under undesirable behaviour during the learning process.
Why this paper is a reference point
Two of the paper’s six co-authors, Dario Amodei and Chris Olah, went on to co-found Anthropic in 2021, alongside Daniela Amodei, Jared Kaplan, Jack Clark, Ben Mann, Sam McCandlish, and Tom Brown. A third co-author, Paul Christiano, led OpenAI’s language model alignment team from 2017 before leaving in 2021 to found the Alignment Research Center. A fourth, John Schulman, was a co-founder of OpenAI in 2015 and later joined Anthropic in 2024 to work on alignment.
That trajectory is part of why the paper is treated as a reference point rather than only a historical document: the specific problems it names — reward hacking, scalable supervision, distributional shift — recur as vocabulary in later technical and governance work, including the kind of lab-level commitments covered in CASRAI’s explainer on responsible scaling policies. The paper itself is a technical research agenda, not a policy framework or a definition of alignment; for that broader context, see CASRAI’s guide to what AI alignment means. CASRAI tracks how these terms are used in practice across frontier labs and governance bodies through NIKOLAI, its structured vocabulary for AI governance and safety terminology.
What the paper does not cover
“Concrete Problems in AI Safety” predates the large language models that dominate current discussion of frontier AI, and its authors are explicit that it sets aside longer-term or more speculative risks in favour of problems that were, at the time, empirically tractable with the machine learning systems then in use. It is a technical paper aimed at machine learning researchers, not a governance or policy document, and it does not propose specific institutions, standards, or oversight bodies. Readers looking for how these ideas have been carried into organisational and governance form should treat it as background vocabulary rather than a current framework.
