CASRAI Dictionary

Tag: evidence synthesis

  • Disclosing AI use in systematic reviews and evidence synthesis

    The systematic review is one of the most demanding forms of scholarship. Its authority rests not on a single clever insight but on a method so transparent that another team could repeat it and reach the same conclusions. Every decision — which databases were searched, with which terms, how records were screened and data extracted, how risk of bias was judged — is meant to be reported in enough detail to be checked. It is precisely this commitment to method that makes the arrival of artificial intelligence in evidence synthesis so consequential. Review teams now use AI and machine-learning tools to help screen thousands of abstracts, classify studies, extract data and even draft text. These tools save effort, but they introduce decisions that must be reported with the same rigour as every other step, or the chain of transparency breaks. This article examines how disclosure norms are forming, drawing on the generative-AI disclosure domain of the CASRAI Dictionary.

    Why AI in evidence synthesis is different

    Disclosing AI use in an ordinary research article is largely about honesty and attribution. In a systematic review the stakes are higher, because the AI is not merely an aid to writing but a participant in the method itself. When a machine-learning classifier helps decide which abstracts are worth full-text screening, it shapes which evidence enters the review and which is excluded. An undisclosed, undocumented automated screening step is a hole in the method through which bias and error can enter unseen. The reader cannot judge a review’s reliability without knowing that part of the screening was automated, which tool was used, how it was configured, and how its decisions were checked. Transparency about AI is therefore not an optional courtesy; it is part of the reproducibility that gives a review its standing.

    PRISMA 2020 and reporting completeness

    The dominant standard for reporting systematic reviews is PRISMA 2020 — the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. PRISMA does not tell authors how to conduct a review; it specifies what they must report so that readers can assess and reproduce it. Its items cover the search, the selection process, the data-collection process and much else, and its flow diagram tracks records from identification through screening to inclusion. The logic of PRISMA maps naturally onto the question of AI: wherever an automated tool participated in identification, screening, data extraction or synthesis, the completeness PRISMA demands implies that this participation be described. The reporting community has been extending this thinking, with guidance and PRISMA-style extensions clarifying how the use of automation tools should appear in the methods and in the flow of records, so that an AI-assisted review is documented to the same standard as a wholly manual one.

    Cochrane and the careful adoption of automation

    Few organisations have engaged with automation in reviews as deeply as Cochrane, whose systematic reviews are a benchmark for the field. Cochrane has cautiously adopted machine-learning tools for tasks such as study classification and screening prioritisation, while insisting they be used in ways that preserve the rigour and transparency reviews require. The consistent themes are instructive: automation may assist human reviewers but should not silently replace human judgement on consequential decisions; a tool’s performance and limitations must be understood; and its use must be reported. Cochrane’s measured approach offers a model for the field, demonstrating that the answer to AI in evidence synthesis is neither prohibition nor uncritical enthusiasm but disciplined, transparent use.

    The RAISE recommendations

    As AI tools have proliferated, the community has worked towards shared recommendations for using them responsibly in evidence synthesis, captured in efforts under the banner of RAISE — responsible AI in evidence synthesis. The thrust of such work is to articulate principles that let reviewers benefit from AI without compromising the integrity of their conclusions. These principles recur across the emerging guidance:

    • Human responsibility. The review team remains accountable for every decision, including those an AI tool assisted; responsibility cannot be delegated to a tool.
    • Transparency of tools. The specific tools used, their versions and how they were configured should be reported, so the method can be understood and repeated.
    • Validation. The performance of an AI tool on the task at hand should be assessed, and its outputs checked, rather than trusted blindly.
    • Clear reporting of role. Exactly which steps the AI participated in — screening, extraction, synthesis — should be stated, so readers know where human and machine judgement met.

    What an AI-assisted review should report

    Drawing these strands together, an evidence synthesis that used AI should make several things plain: it should name the tools and their versions and describe what each was used for; explain how each was configured and, where relevant, trained or calibrated; describe how its decisions were checked, such as whether a sample of automated screening decisions was verified by human reviewers; and be honest about limitations. None of this requires abandoning AI; it requires treating AI exactly as the method demands every other step be treated — described fully enough to be judged and repeated. A review that conceals its use of automation forfeits the very transparency that distinguishes evidence synthesis from mere opinion.

    A consistent vocabulary for AI disclosure

    For disclosures of this kind to be useful across journals, databases and the systems that index reviews, what is being disclosed has to be described consistently — which tool, used for which task, checked in which way. That consistency is what the CASRAI Dictionary works towards: a shared vocabulary so that a statement about AI use in a review means the same thing wherever it is recorded. And because conducting a systematic review is substantial, recognisable scholarly work — searching, screening, extracting, appraising, synthesising — the contributions behind it can be described in the same shared framework, the CRediT taxonomy and its full set of contribution roles, which sits within the broader practice of research administration. AI will keep changing how evidence is synthesised; the enduring obligation — to report the method fully and honestly — is what keeps a review trustworthy.

  • Crediting contributions in systematic reviews and meta-analyses

    A systematic review looks, from the outside, like a single coherent document with a tidy list of authors. From the inside it is a small project with a remarkable division of labour: a protocol to register, a search strategy to design and run across multiple databases, thousands of records to screen against eligibility criteria, full texts to retrieve and assess, data to extract twice over, risk-of-bias judgements to make, a synthesis or meta-analysis to compute, and a report to write to an exacting standard. Each of those tasks is a distinct skill, and each is usually done by a different person or pair of people. The conventional author byline flattens all of it. This article looks at how structured reporting through PRISMA and structured contributorship through the CRediT taxonomy together make the real shape of this work visible, and where the vocabulary for it sits in the credit extensions domain of the CASRAI Dictionary.

    Why a review is hard to credit fairly

    The difficulty is that the most laborious and methodologically critical parts of a review are precisely the ones that leave no trace in a traditional byline. Screening twenty thousand abstracts in duplicate is exacting, consequential work — get the eligibility judgements wrong and the whole review is compromised — yet it is invisible in author order. The same is true of designing a reproducible search, performing duplicate data extraction, or making risk-of-bias assessments. Meanwhile, the person who conceived the question and the person who drafted the manuscript are easy to recognise. A fair account of a review has to name the unglamorous, high-stakes tasks as clearly as the visible ones.

    PRISMA: reporting the process transparently

    The first half of the answer is methodological transparency. PRISMA — Preferred Reporting Items for Systematic Reviews and Meta-Analyses — is the reporting guideline that tells readers what a review actually did: how the search was constructed, how records moved from identification through screening to inclusion (the familiar flow diagram), how data were extracted, and how studies were appraised and synthesised. PRISMA does not assign credit, but it makes the work auditable. When a review reports its process to the PRISMA standard, the existence and scale of each task — the searching, the screening, the extraction, the appraisal — becomes explicit rather than implied. That visibility is the precondition for crediting it: you cannot recognise a contribution that the reporting has hidden.

    CRediT: naming who did what

    The second half is contributorship. The Contributor Roles Taxonomy provides a controlled vocabulary of contribution types that maps unusually well onto the anatomy of a review. The full set is set out in our overview of the CRediT roles, but several are worth singling out for evidence synthesis:

    • Conceptualization — formulating the review question and eligibility criteria.
    • Methodology — designing the search strategy and the synthesis approach, often the work of an information specialist.
    • Investigation — running the searches, screening records and retrieving full texts.
    • Data curation — managing the extracted data, de-duplication and the records that underpin the flow diagram.
    • Formal analysis — the meta-analysis itself, including heterogeneity assessment and any sensitivity analyses.
    • Writing – original draft and Writing – review & editing — producing and refining the manuscript.

    Used together, these roles let a review record that the information specialist designed the search, that two named reviewers screened and extracted in duplicate, and that the statistician ran the synthesis — rather than leaving all of it to be guessed from author order. The wider CRediT taxonomy turns the division of labour into a machine-readable statement attached to the output.

    The role information specialists deserve

    One contribution that systematic reviews chronically under-credit is that of the information specialist or research librarian who designs and validates the search. A poorly constructed search undermines a review more surely than almost any other flaw, and a well-constructed one is a genuine methodological achievement. Recording this work explicitly under Methodology and Investigation — rather than relegating it to an acknowledgement — is one of the clearest practical gains from applying contributorship to evidence synthesis. It names a contribution that is both critical and routinely invisible.

    Crediting duplicate work without double-counting

    Reviews rely on tasks done independently by two people — duplicate screening, duplicate extraction — precisely to reduce error. Contributorship should reflect that both reviewers did the work, which CRediT handles naturally by allowing a role to be assigned to more than one contributor. The honest principle, as ever, is that a role records what a person actually did: both screeners earn the Investigation role because both genuinely screened, not as a courtesy. This is the same standard that applies across all contribution recording — credit follows real work, and is neither inflated for visibility nor withheld for convenience.

    A consistent record across systems

    Systematic reviews increasingly register protocols, deposit search strategies and data, and publish in journals that require both PRISMA reporting and a contributorship statement. For that ecosystem to work, the way a contribution is described has to mean the same thing wherever it appears. That consistency is what the CASRAI Dictionary exists to provide: a stable vocabulary so that a Methodology contribution declared in a protocol registry, a manuscript and an institutional record can be recognised as the same claim. Combined with PRISMA’s transparency about process, structured contribution makes the substantial, distributed work of evidence synthesis legible — crediting the screeners, extractors and search designers whose labour holds a review together, not only the names at the top of the list.