On November 10, 2026, researchers from the FAIR Data Innovations Hub and the California Digital Library released an arXiv preprint, Evaluating the Performance of LLMs in Drafting NIH Data Management Plans. The team tested large language models on drafting NIH Data Management and Sharing plans across required elements. Although commercial models accelerated initial drafting, data librarians found omissions in repository details, proving artificial intelligence cannot replace expert institutional review.
What happened
A research team led by Nahid Zeinali, James O’Neil, Xuebin Dong, Rebecca Hofstein Grady, Maria Praetzellis, Brian Riley, John Chodacki, and Bhavesh Patel evaluated how effectively frontier large language models draft funder-compliant data management and sharing plans. Representing the FAIR Data Innovations Hub at the California Medical Innovations Institute and the California Digital Library at the University of California Office of the President, the authors compared two architectures: OpenAI’s commercial GPT-4.1 and Meta’s open-source Llama 3.3 70B, which ran locally using Ollama on dual RTX 3090 GPUs.
The researchers benchmarked both models against 26 sample plans published by the National Institutes of Health under the NIH Data Management and Sharing Policy. These reference plans covered eight NIH institutes and 21 research tags, including clinical trials, genomics, imaging, and behavioral studies. Using a minimal-prompt approach, the models received the target institute, human-subject consent type, and brief project details from Element 1A, then drafted all six required elements and 12 sub-elements of the NIH template.
Methodology and key findings
The evaluation combined automated reference-based text analysis with blinded review by 21 domain experts. In automated comparisons across all 26 reference plans, GPT-4.1 scored higher than Llama 3.3 in semantic similarity via SBERT (0.64 versus 0.61; p = 6.6 × 10⁻⁸) and lexical overlap via ROUGE (0.28 versus 0.26; p = 9.7 × 10⁻⁵). The performance gap proved widest in technical sections, specifically repository selection (Element 4A) and data access considerations (Element 5A).
The human panel comprised 21 specialists, including 12 research data librarians (57.1%) and four data stewards (21.4%), with 76.2% possessing four or more years of experience. Evaluators reviewed 10 representative plans across three blinded versions: the NIH reference sample, a GPT-4.1 draft, and a Llama 3.3 draft.
Overall satisfaction favored GPT-4.1: 90.5% of expert ratings were satisfied or very satisfied (mean element ratings of 3.9 to 4.4 out of 5), compared to 42.9% for human-written NIH samples and 19.0% for Llama 3.3. In blinded classification, 85.7% of reviewers mistook GPT-4.1 drafts for human-authored text, whereas 71.4% recognized Llama 3.3 as machine-generated. However, evaluators identified 48 errors across GPT-4.1 plans, primarily vague repository designations and incomplete data sharing timelines.
Why it matters for research practice
The findings illustrate both the immediate utility and technical limits of generative artificial intelligence in research administration. Commercial models master administrative tone and formatting, but off-the-shelf open-source models struggle with precision when naming repositories or metadata standards without fine-tuning or retrieval augmentation.
This dynamic directly affects institutional infrastructure such as the DMPTool, managed by co-authors at the California Digital Library. Funder compliance requires alignment with FAIR data principles, ensuring data is findable, accessible, interoperable, and reusable in established repositories. When models supply generic placeholders rather than qualified repositories, proposals face compliance risks during review.
What changes for research data librarians and sponsored projects
For sponsored projects offices and libraries, the study confirms that language models function as drafting aids rather than autonomous authors. Researchers consulting an NIH data management and sharing plan template can use language models to draft baseline text, but data librarians must verify repository selections, access restrictions, and institutional resources before grant submission.
The authors note the NIH has prepared simplified plan guidelines with structured questions for implementation in late May 2026. While structured templates may reduce drafting errors, professional data curation remains critical for ethical data governance and policy compliance.
Sources
- Nahid Zeinali, James O’Neil, Xuebin Dong, Rebecca Hofstein Grady, Maria Praetzellis, Brian Riley, John Chodacki, and Bhavesh Patel. Evaluating the Performance of LLMs in Drafting NIH Data Management Plans. arXiv:2611.14648v1, submitted November 10, 2026.
- National Institutes of Health. Final NIH Policy for Data Management and Sharing. Notice NOT-OD-21-013.
- FAIR Data Innovations Hub. nih-dmp-llm-evaluation-paper-data. Dataset archived on Zenodo, November 2026.







