Written and maintained by CASRAI Editorial Board
Last updated
Bias in an experiment is a systematic error: a consistent push in the result away from the true value, caused by how the study was designed, run, measured or reported. It is not the same as chance. This guide maps each source of bias to the design feature that guards against it, using the current Cochrane risk-of-bias domains, and shows the arithmetic with a worked example.
What is bias in an experiment?
Cochrane’s Handbook defines bias as “a systematic error, or deviation from the truth, in results.” If the same experiment were repeated many times, a biased design would miss the true value in the same direction on average. A larger sample does not repair that, because every extra participant carries the same distortion.
The practical question is therefore not “is there any error?” (there always is) but “does the error lean one way, and which step of the design makes it lean?” The single-bias pages on this site cover individual mechanisms in depth, for example selection bias and measurement bias; this page is the map that connects them to the safeguards.
Bias versus random error
The Handbook states that “bias should not be confused with imprecision”, and describes imprecision as random error. The two behave differently:
- Random error scatters results around the truth. More participants and repeated measurement shrink it.
- Bias shifts the centre of the results away from the truth. More participants only make the wrong answer look more certain.
A related point: a baseline difference between randomised groups that arises by chance is not bias. The Handbook states that “chance imbalances are not a source of systematic bias”, and that baseline imbalances consistent with chance are not read as evidence of risk of bias. What matters is whether the allocation process itself could have been manipulated or predicted.
Formula: the bias of an estimate
In statistics the bias of an estimator is the gap between its expected value and the true value:
bias(θ̂) = E[θ̂] − θ
- θ (theta) is the true value of the quantity you are trying to estimate, such as a true treatment effect.
- θ̂ (theta-hat) is the estimate your experiment produces.
- E[θ̂] is the expected value of that estimate: the average you would get if you could repeat the whole experiment indefinitely.
Zero bias means the estimate is centred on the truth. It says nothing about spread, which is measured separately by the standard deviation (SD) or standard error. The true value θ is never known in a real study, so this formula is used in simulation, method checking and teaching; in real studies, bias is judged from the design.
The five sources of bias (Cochrane RoB 2 domains)
The Cochrane Handbook organises risk of bias for randomised trials into five domains. The table gives the exact domain names, the older textbook label most people search for, and the design safeguard usually aimed at each.
| RoB 2 domain | Older label | Typical safeguard | Read more |
|---|---|---|---|
| Bias arising from the randomization process | Selection bias | Unpredictable sequence generation, concealed allocation | Randomization methods |
| Bias due to deviations from intended interventions | Performance bias | Blinding of participants and those delivering the intervention | Blinding and masking |
| Bias due to missing outcome data | Attrition bias | Keeping participants in follow-up; recording reasons for loss | RoB 2 traffic-light plot |
| Bias in measurement of the outcome | Detection bias | Blinded or standardised outcome assessment | Concealment vs blinding |
| Bias in selection of the reported result | Reporting bias | A pre-specified analysis plan, followed and disclosed | RoB 2 traffic-light plot |
Two points about the mapping. First, the domains follow where in the study the problem enters, not a list of named biases; the Handbook notes that recent tools focus on risk of bias rather than the broader notion of “methodological quality”. Second, the Handbook observes that errors in measuring outcomes can bias intervention effect estimates, which is why the outcome-measurement domain is separate from the allocation domain. Concealing the allocation sequence and blinding people after allocation are different safeguards aimed at different domains.
Confounding is a related but distinct threat in non-randomised designs; see confounding variable.
Worked example with the arithmetic shown
Illustrative numbers, computed for this guide; not data from a real study. Suppose the true effect of an intervention on systolic blood pressure is a 5.0 mmHg reduction, so θ = 5.0. Five hypothetical repeats of an experiment give the estimates below under two designs.
| Repeat | Design A (assessor unblinded, leans toward benefit) | Design B (blinded, small sample, noisy) |
|---|---|---|
| 1 | 7.1 | 2.9 |
| 2 | 6.4 | 6.6 |
| 3 | 7.8 | 4.1 |
| 4 | 6.9 | 5.8 |
| 5 | 6.8 | 5.6 |
| Sum | 35.0 | 25.0 |
| Mean, E[θ̂] estimated | 35.0 ÷ 5 = 7.0 | 25.0 ÷ 5 = 5.0 |
| Bias = mean − θ | 7.0 − 5.0 = +2.0 | 5.0 − 5.0 = 0.0 |
| Sample SD | 0.51 | 1.48 |
SD for Design A: squared deviations from 7.0 are 0.01, 0.36, 0.64, 0.01 and 0.04, which sum to 1.06; 1.06 ÷ 4 = 0.265; the square root is 0.51. SD for Design B: squared deviations from 5.0 are 4.41, 2.56, 0.81, 0.64 and 0.36, which sum to 8.78; 8.78 ÷ 4 = 2.195; the square root is 1.48.
Interpretation: how to read the result
Design A is precise but wrong: its results cluster tightly around 7.0, a full 2.0 mmHg above the truth. Design B is imprecise but unbiased: any single run can miss badly, yet the average lands on the truth. Reporting only the spread would make Design A look like the better study.
| Bias | Spread (SD) | What it means |
|---|---|---|
| Near zero | Small | Estimates are accurate and consistent |
| Near zero | Large | Right on average, but any single result is unreliable; increase sample size |
| Clearly non-zero | Small | Confidently wrong; fix the design, because more data will not help |
| Clearly non-zero | Large | Both problems |
Assumptions and common mistakes
- Treating a bigger sample as a cure. Sample size reduces random error only.
- Calling every baseline difference bias. Chance imbalance after proper randomisation is not systematic bias, as noted above.
- Confusing concealment with blinding. The first protects the allocation step; the second protects what happens after it. Compare them in allocation concealment vs blinding; the comparison group itself is covered in control group.
- Assuming bias can be measured in a real study. It needs the true value, which you do not have. You assess the risk of bias from the design and conduct.
- Judging by the listed name of a bias rather than the mechanism. Ask which step of the design could push results one way.
How to report it
Report the safeguards you used, step by step, rather than a general claim of “no bias”. An APA-style methods sentence might read: “Participants were randomly allocated by a computer-generated sequence held by an independent coordinator, and outcome assessors were unaware of group allocation.” Cite the guidance you followed (for example the Cochrane Handbook chapters listed under Sources) in the reference list, in the format your journal or style guide requires.
How to calculate bias in Excel, R or Python
These snippets reproduce Design A from the worked example, where the true value is known (a simulation or method check).
Excel: with the estimates in cells B2:B6 and the true value 5 in cell E1, bias is =AVERAGE(B2:B6)-E1 and the spread is =STDEV.S(B2:B6).
R:
est <- c(7.1, 6.4, 7.8, 6.9, 6.8)
theta <- 5.0
mean(est) - theta # 2.0
sd(est) # 0.5148Python:
import statistics as st
est = [7.1, 6.4, 7.8, 6.9, 6.8]
theta = 5.0
print(round(st.mean(est) - theta, 2)) # 2.0
print(round(st.stdev(est), 4)) # 0.5148Frequently asked questions
Is bias the same as random error?
No. Cochrane describes bias as systematic error and imprecision as random error. Random error scatters results around the truth; bias moves their centre.
Does a larger sample remove bias?
No. A larger sample narrows random error, but a systematic distortion built into the design is repeated in every participant.
Is a difference between groups at baseline a sign of bias?
Not by itself. After proper randomisation, baseline differences that are consistent with chance are not interpreted as evidence of risk of bias.
Sources
- Cochrane Handbook for Systematic Reviews of Interventions, Chapter 7 (bias and imprecision definitions): https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07 (checked 2026-10-29)
- Cochrane Handbook for Systematic Reviews of Interventions, Chapter 8 (RoB 2 domains; chance imbalances; outcome measurement): https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08 (checked 2026-10-29)
