Peer reviewing a systematic review or meta-analysis is not the same task as peer reviewing an original-research manuscript, and reviewers who apply original-research habits to a review submission tend to miss the errors that matter most. An original-research review asks whether a single study’s design and analysis support its own conclusions. Reviewing a review asks a different question: did the authors search, select, appraise, and synthesize the existing literature in a way that is complete, reproducible, and free of avoidable bias? The evidence itself was generated by other people’s studies; the reviewer’s job is to judge the rigor of the process the authors used to find and combine it.
This guide sets out what a peer reviewer specifically checks when evaluating a systematic-review or meta-analysis submission, organized around the three areas where review-type manuscripts most often fail: reporting compliance, search-strategy completeness, and risk-of-bias/quality-appraisal rigor. It complements CASRAI’s broader peer reviewer guide and peer review overview, which cover reviewer roles and review models generally rather than review-type manuscripts specifically.
How reviewing a review differs from reviewing original research
In an original-research manuscript, a reviewer typically assesses one study’s hypothesis, sample, methods, and statistics. In a systematic review or meta-analysis, the “sample” is the body of literature itself, and the methodological choices under scrutiny are about literature retrieval and evidence synthesis rather than data collection. A reviewer needs to check:
- Whether the search was comprehensive and reproducible enough that another team could run it again and get essentially the same result.
- Whether study selection and data extraction were done in a way that limits reviewer bias (independent dual screening, documented conflict resolution).
- Whether the risk of bias in the included studies was assessed with a tool appropriate to their design, and whether that assessment actually influenced how results were interpreted or weighted.
- Whether any statistical synthesis (meta-analysis) was appropriate given the heterogeneity of the included studies, rather than mechanically pooling data that shouldn’t have been pooled.
- Whether the manuscript reports against a recognized checklist, most commonly PRISMA 2020, so that readers can verify each of the above themselves.
Check reporting compliance against PRISMA 2020
PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, Page MJ et al., BMJ 2021;372:n71) is a 27-item reporting checklist, not a methodology for conducting a review — a manuscript can follow PRISMA’s reporting structure and still have used weak methods underneath it. A reviewer’s job on this front is narrower than it sounds: confirm the authors have actually reported what the checklist requires, item by item, and flag where reporting is vague or missing rather than simply present.
In practice this means checking that the manuscript includes:
- A PRISMA flow diagram documenting how many records were identified, screened, excluded (with reasons for full-text exclusions), and ultimately included — checklist item 16a explicitly asks for this, “ideally using a flow diagram.”
- A full, reproducible search strategy (see below), not just “we searched PubMed and Google Scholar.”
- A stated eligibility/PICO(S) framework defined before screening began, rather than criteria that appear to have been shaped to fit whichever studies were found.
- Explicit description of the risk-of-bias tool used and how its results fed into the synthesis or discussion, per PRISMA’s risk-of-bias-reporting items.
- A synthesis methods section that states how heterogeneity was handled, not just a pooled effect estimate with no discussion of how consistent the underlying studies actually were.
CASRAI’s PRISMA and systematic review methodology guide covers the checklist and flow diagram in full and is the reference to point authors to when a review’s reporting is incomplete rather than merely imperfect.
Assess search-strategy completeness and reproducibility
A systematic review’s credibility rests on whether its search actually found the relevant literature — this is the single most reviewer-checkable determinant of quality, because unlike risk-of-bias judgment calls, a search strategy is either reproducible or it isn’t. A reviewer should be able to answer:
- Are the full search strings provided? Boolean strings for at least one database, ideally as a supplementary file, with enough detail that a librarian or another reviewer could rerun it and expect a comparable result set.
- Were multiple databases searched? A single-database search (PubMed alone, for example) is a common and reviewable weakness; most fields expect at least two or three major databases appropriate to the discipline.
- Was grey literature considered? Trial registries, conference proceedings, dissertations, and preprint servers matter specifically because omitting them skews a synthesis toward published, positive results — PRISMA 2020 requires reviewers to search trial registries and grey literature specifically to reduce this kind of reporting bias, not just indexed journal databases.
- Is the search date reported, and is it recent enough relative to submission that the review isn’t presenting an already-stale evidence base?
- Was the protocol registered in advance? Prospective registration — most commonly in PROSPERO, the International Prospective Register of Systematic Reviews — lets a reviewer check whether the eligibility criteria, outcomes, and analysis plan were fixed before the authors saw the results, or changed afterward to fit them. A reviewer who can find the registered protocol should compare it against the submitted manuscript and flag undisclosed deviations.
Evaluate risk-of-bias and quality-appraisal rigor
This is where reviewing a review most departs from reviewing original research: the reviewer is checking someone else’s appraisal of other studies’ bias, not appraising a single study’s own design directly. Two things need checking together, and manuscripts frequently get only one right:
- Was the right tool used for the included study designs? Cochrane’s RoB 2 is built for randomized trials; ROBINS-I is the equivalent for non-randomized studies of interventions. A review that applies an RCT-oriented tool to observational studies (or vice versa) has used the wrong instrument, and its risk-of-bias judgments are unreliable regardless of how carefully they were applied.
- Was the tool applied consistently and, ideally, independently by two reviewers with documented disagreement resolution — the same dual-reviewer logic that applies to study screening should apply to risk-of-bias judgments, since these are judgment calls, not mechanical extractions.
- Did the risk-of-bias assessment actually change anything? A common failure mode is a risk-of-bias table that’s present but disconnected from the rest of the manuscript — high-risk studies pooled with low-risk ones on equal footing, or a “limitations” paragraph that doesn’t reflect what the table actually found. A reviewer should check whether sensitivity analyses excluding high-risk studies were run, and whether certainty was downgraded accordingly.
Two further, related tools a reviewer may encounter or should recommend:
- AMSTAR 2 — a 16-item tool for critically appraising the methodological quality of a systematic review itself (as opposed to appraising the studies inside it). Editors and reviewers sometimes use AMSTAR 2 as a structured checklist for evaluating the submission as a whole.
- ROBIS — a tool specifically for assessing risk of bias in a systematic review (as distinct from bias in its included studies), completed in three phases: assessing relevance, identifying concerns with the review process across four domains, and judging overall risk of bias in the review’s conclusions.
- GRADE — used to rate the overall certainty of a pooled estimate (high, moderate, low, very low) across five domains including risk of bias, inconsistency, indirectness, imprecision, and publication bias; a reviewer should check whether a review presenting a meta-analysis has graded certainty at all, not just reported a point estimate and confidence interval.
Check the meta-analysis, if there is one
Not every systematic review includes a statistical meta-analysis, and a reviewer’s first question should be whether pooling was appropriate at all. Specific checks:
- Is clinical and methodological heterogeneity across the included studies low enough to justify pooling, or have dissimilar populations/interventions/outcomes been combined into a single effect estimate that obscures more than it reveals?
- Is statistical heterogeneity (I², Cochran’s Q) reported and interpreted, and does a random-effects vs. fixed-effect model choice match what that heterogeneity implies?
- Was publication bias assessed (e.g., funnel plot, Egger’s test) where the number of included studies is large enough for that assessment to be meaningful, and is its inherent low power with few studies acknowledged rather than overstated as a clean finding?
- If subgroup or sensitivity analyses are reported, were they pre-specified in the protocol, or introduced post hoc — post hoc subgroup analysis in a meta-analysis carries the same interpretive caution as post hoc subgroup analysis in a single trial.
Frequently asked questions
Do I need to re-run the search myself as a reviewer?
No — a reviewer isn’t expected to independently re-execute the search strategy. The check is whether the search is reproducible (full strings, databases, dates, and grey-literature sources all reported), not whether the reviewer personally reproduces it.
What if the review doesn’t follow PRISMA at all?
Most journals that publish systematic reviews require or strongly recommend PRISMA 2020 compliance at submission. If a manuscript doesn’t reference PRISMA and lacks a flow diagram or reproducible search strategy, that’s a substantive methodological gap to raise with the editor, not a stylistic preference to note in passing.
Is reviewing a meta-analysis different from reviewing a systematic review without one?
The search-strategy and risk-of-bias checks above apply to both. A meta-analysis adds a further layer: the statistical appropriateness of pooling, covered in the “Check the meta-analysis” section above. A systematic review that stops at a narrative synthesis (no pooling) doesn’t need that layer evaluated, but does still need synthesis-method reporting checked against PRISMA.
Which risk-of-bias tool should I expect for a review of observational studies?
ROBINS-I is the standard tool for non-randomized studies of interventions; RoB 2 is for randomized trials specifically. A review that mixes both study designs should report using the appropriate tool for each subset, not force one tool across both.
