Data management plans have become a near-universal requirement. Funders ask for them at the proposal stage, institutions increasingly expect them, and researchers have largely accepted that planning for data is part of designing a project. But requiring a plan and getting a good plan are two very different things. A DMP written hastily to satisfy a requirement, glanced at once and never looked at again, achieves almost nothing — it is a box ticked, not a commitment made. The harder, less-discussed half of the DMP story is evaluation: how plans are actually reviewed, against what criteria, by whom, and with what consequences. As DMPs mature, attention is rightly shifting from whether they exist to whether they are any good. This article examines DMP evaluation, drawing on the machine-actionable DMP domain of the CASRAI Dictionary.
Why evaluation matters
The case for taking DMP review seriously is straightforward. If a plan is never assessed, there is little incentive to write a good one, and the requirement degenerates into a formality that consumes effort without improving practice. Evaluation is what gives a DMP teeth: it signals that the plan is expected to be substantive, it provides researchers with feedback they can act on, and it lets funders identify proposals where the data-handling arrangements are inadequate or unrealistic. A reviewed DMP is a commitment someone has engaged with; an unreviewed DMP is a wish.
Rubrics and review criteria
To review plans fairly and consistently, reviewers need criteria, and this has driven the development of DMP rubrics — structured frameworks that lay out what a good plan should address and how to judge it. A rubric breaks the assessment down into components and gives reviewers a consistent basis for judging each one, so that plans are evaluated against the same expectations rather than according to each reviewer’s personal sense of what matters. Typical dimensions a rubric covers include:
- Data description. Is it clear what data will be produced or used, in what formats and volumes?
- Documentation and metadata. Will the data be documented well enough to be understood and reused?
- Storage and security. Are arrangements for storing and protecting the data, including any sensitive data, adequate?
- Preservation and sharing. Where will the data be deposited, under what access conditions and licence, and for how long?
- Ethical and legal compliance. Are consent, privacy and legal obligations properly addressed?
- Roles and resources. Is it clear who is responsible, and are the resources to do this realistic?
One prominent example is the DART (Data management plan Analysis, Reporting and Tracking) rubric, developed to help institutions and reviewers assess DMPs systematically and consistently. Tools and rubrics of this kind matter because they turn “is this a good plan?” — a vague and subjective question — into a structured assessment that different reviewers can apply in comparable ways.
Funder assessment in practice
Funders approach DMP assessment in different ways and at different points. Some review the plan as part of the proposal, treating the quality of data-handling arrangements as one factor in deciding what to fund. Others emphasise the DMP as a project deliverable, expecting it to be developed and updated as the project proceeds. In either case, the trend is towards taking the plan seriously as something to be engaged with, not merely collected. There is a balance to strike: assessment should be rigorous enough to improve practice but proportionate enough not to impose a heavy burden. A purely bureaucratic review risks producing better-written but no better-managed data; the aim is to improve what actually happens to the data, not just the prose describing it.
Feedback loops
Perhaps the most valuable, and most often neglected, aspect of DMP evaluation is the feedback loop. Assessment is most useful when it is not merely a gate — pass or fail — but a source of guidance that helps researchers improve their plans and their practice. Feedback can flow in several directions:
- To the researcher, pointing out weaknesses and suggesting improvements, ideally early enough to act on.
- Into the project, where a plan reviewed at the start can be revisited and updated as the work develops and the data takes shape.
- Back to support services, where patterns across many plans reveal where researchers commonly struggle, so that training and support can be targeted.
Feedback is what turns evaluation from a judgement into a constructive tool. A plan that comes back with specific, actionable comments helps the researcher do better; a plan that simply passes or fails teaches nothing.
Machine-actionable checks
The move towards machine-actionable DMPs (maDMPs) opens a powerful possibility for evaluation: automating the parts of review that can be automated. When a plan is expressed as structured, machine-readable data rather than free prose, certain checks no longer require a human. A system can verify whether a repository has been specified, whether a licence has been chosen, whether an identifier has been minted, or whether commitments are consistent with funder policy. This does not replace expert human judgement — assessing whether the chosen approach suits the research still requires understanding — but it can handle the routine, checkable elements automatically, freeing reviewers to focus on the judgements that genuinely need them. Machine-actionable checks can also run continuously, so that a living plan is monitored against its commitments throughout a project rather than assessed only once.
A shared vocabulary for review
For DMP evaluation to work consistently — across funders, institutions and the tools that support planning — the elements being reviewed and the criteria applied must mean the same thing everywhere. A plan written against one set of expectations and reviewed against another, or described in terms a reviewing system cannot interpret, defeats the purpose. That consistency is what the CASRAI Dictionary supports: a shared vocabulary so that the components of a data management plan are understood identically by those who write them and those who review them, supporting sound research administration. And because reviewing and supporting data management is genuine contribution, the work can be described in the same framework used for every other — the CRediT taxonomy and its full set of contribution roles. A DMP is only as valuable as the seriousness with which it is reviewed; good evaluation is what turns the plan from a promise into a practice.