Differential Item Functioning (DIF): Detecting Item-Level Bias with the Mantel-Haenszel Method

Differential item functioning (DIF) occurs when a single item on a test or scale produces a different probability of endorsement or a correct response for people from different groups who are genuinely at the same level of the underlying trait. That “same level” qualifier is what separates DIF from an ordinary group difference in test scores. Two groups scoring differently on a whole scale is not, by itself, evidence that anything is wrong with the instrument — it may simply mean the groups differ on the trait being measured. DIF is narrower and more diagnostic: it asks whether one specific item behaves inconsistently for people who have already been matched on ability, which is a property of that item, not of the groups. This guide covers how DIF is distinguished from overall scale bias and from ordinary group differences (“impact”), the difference between uniform and non-uniform DIF, and the Mantel-Haenszel procedure most commonly used to detect it — with a fully reproducible worked example.

DIF vs. Test Bias vs. Impact

Three related but distinct ideas get conflated constantly in discussions of test fairness, and keeping them separate is most of what makes DIF analysis useful rather than alarming:

• Impact is the raw, unconditional difference in mean scores, pass rates, or endorsement rates between groups on a whole test or scale. Impact can be entirely legitimate — if the groups genuinely differ on the trait being measured, their scores should differ too — or it can be a downstream symptom of item-level problems. Impact alone doesn’t distinguish the two.
• Test (or scale) bias is a claim about the whole instrument: that it systematically measures something construct-irrelevant differently across groups, in a way that distorts the score’s meaning for one group relative to another.
• DIF is item-level and conditional. It asks whether, among test-takers matched on the trait the test is trying to measure — typically approximated by total score on the rest of the instrument, or a theta estimate from an IRT model — the reference and focal group have different probabilities of the keyed response on this one item. A test can be entirely free of DIF and still show impact (real group differences in the trait); a test can also show DIF on individual items that happen to cancel out, producing little or no net impact at the total-score level.

This three-way distinction is standard in the measurement-fairness literature (see, e.g., Camilli & Shepard’s Methods for Identifying Biased Test Items, and Osterlind & Everson’s Sage QASS volume Differential Item Functioning) and is treated as part of the evidence base for fairness in the joint AERA/APA/NCME Standards for Educational and Psychological Testing. DIF analysis doesn’t replace judgment about what the construct should mean for different groups — it flags candidates for that judgment. For the broader validity framework DIF sits inside, see Construct Validity and Reliability in Research Measurement.

Uniform vs. Non-Uniform DIF

Holland and Thayer’s foundational 1988 treatment of the Mantel-Haenszel procedure for DIF distinguishes two patterns, and the distinction matters for which detection method will actually catch the problem:

• Uniform DIF: one group has a consistent advantage on the item across the entire trait range. In item response theory terms, this corresponds to the item’s difficulty (location) parameter differing by group while discrimination stays the same — the item characteristic curves are shifted relative to each other but never cross.
• Non-uniform DIF: the direction or size of the advantage changes across the trait range — low-trait members of one group might be favored while high-trait members of the other group are favored, or the gap narrows or reverses. This corresponds to a difference in the item’s discrimination parameter, so the item characteristic curves cross somewhere in the range.

The ordinary Mantel-Haenszel statistic is built to detect uniform DIF; it assumes the odds ratio between groups is constant across strata and pools accordingly. A genuinely non-uniform pattern can partly cancel across strata in that pooled estimate, understating or missing the effect — which is why non-uniform DIF is usually screened for with a different or supplementary method (logistic regression DIF, or an IRT-based likelihood ratio test), covered briefly below.

The Mantel-Haenszel DIF Procedure

The Mantel-Haenszel (MH) approach, adapted for DIF by Holland and Thayer (1988) from the general Mantel-Haenszel method for pooling stratified 2×2 tables, is the most widely used DIF-detection method because it needs no distributional assumptions about the trait and works with ordinary total-score matching rather than a fitted IRT model. The procedure:

1. Score the item dichotomously (0/1 — correct/incorrect, or endorsed/not endorsed).
2. Designate a reference group and a focal group in advance, before looking at the results — not after seeing which group scored lower on the item.
3. Choose a matching variable that approximates the trait: total score on the rest of the test (an anchor set) is standard. The more defensible practice excludes the studied item itself from that anchor score, so the item isn’t used to help define the very stratum it’s then tested within; simplified treatments sometimes include it, which is a smaller distortion on a long test but worth avoiding on short instruments.
4. Stratify respondents into score groups (strata) on the matching variable, then build a 2×2 table within each stratum: group (reference/focal) by item response (correct/incorrect).
5. Pool across strata into the MH common odds ratio, using cell counts A (reference-correct), B (reference-incorrect), C (focal-correct), D (focal-incorrect), and stratum total T_k:

α_MH = Σ(A_kD_k / T_k) ÷ Σ(B_kC_k / T_k)

An α_MH of 1.0 means no association between group and item response after conditioning on the matching variable — no DIF. Values above 1.0 indicate the reference group has higher odds of the keyed response at matched trait levels (an item favoring the reference group); values below 1.0 indicate the reverse.

6. Convert to the delta (MH D-DIF) scale, the convention ETS uses so effect sizes are reported on the same metric as item-difficulty deltas:

MH D-DIF = −2.35 × ln(α_MH)

The sign flips the direction so a negative MH D-DIF favors the reference group and a positive value favors the focal group, with 0 meaning no DIF.

7. Test significance with the continuity-corrected MH chi-square (1 df):

χ²_MH = (|Σ(A_k − E[A_k])| − 0.5)² ÷ ΣVar(A_k)

8. Classify effect size, not just significance. With a large enough sample, a trivial α_MH close to 1.0 can still test as “significant” — which is why ETS’s widely used convention pairs the significance test with a magnitude threshold on MH D-DIF, roughly: negligible (Category A, |D-DIF| under about 1.0 or not statistically significant), moderate (Category B, |D-DIF| significant and in the 1.0–1.5 range), and large (Category C, |D-DIF| significant and 1.5 or above). Treat the exact cutpoints as a widely cited convention rather than a fixed statistical law — different testing programs adapt the thresholds to their own stakes.

Worked Example: A Simulated Item With and Without True DIF

This example uses simulated data, generated and computed for this guide with a seeded random-number generator so the results are exactly reproducible — it is not drawn from a real study. 1,000 simulated test-takers (500 reference, 500 focal) were drawn from the same trait distribution — theta ~ N(0,1) in both groups — so that any DIF the analysis finds is attributable to the item’s own behavior, not to a real group difference in the trait. Each simulated respondent answered a 9-item anchor set (identical item parameters for both groups, no DIF, used purely as the matching variable) under a two-parameter logistic (2PL) IRT model, plus a 10th, “studied” item deliberately given a true uniform DIF: the same discrimination for both groups, but a difficulty parameter 0.8 logits higher for the focal group (b_ref = 0.0, b_focal = 0.8) — meaning a focal-group respondent needs a higher trait level than a reference-group respondent to have the same probability of answering correctly. A control item with identical parameters for both groups (b_ref = b_focal = 0.2, no true DIF) was run through the identical pipeline to show what a clean result looks like.

Stratifying on the 9-item anchor score (0–9) produced these 2×2 tables for the studied item:

Pooling those ten strata into the MH statistics:

Item	αMH	MH D-DIF	χ²MH (1 df)	Approx. p	ETS category
Studied item (true DIF injected)	2.65	−2.29	47.9	<.001	C (large, favors reference group)
Control item (no DIF injected)	1.17	−0.38	1.28	≈.26	A (negligible)

The procedure recovers the injected effect cleanly: the item that was deliberately made harder for the focal group at matched trait levels comes back with a large, highly significant, reference-favoring MH D-DIF (−2.29, Category C), while the item with identical parameters for both groups comes back statistically unremarkable (Category A, p ≈ .26) — exactly the null result a well-behaved item should produce. Note the sign: because α_MH > 1 here means the reference group has higher odds of a correct response at matched ability, the resulting MH D-DIF is negative, per the reference-favors-negative convention above.

What a Flagged Item Actually Means — and What to Do About It

A statistically flagged item is a candidate for review, not a verdict. The MH procedure (or any DIF method) tells you an item behaves differently for matched groups; it cannot by itself tell you why. Two broad explanations exist, and only one of them is a fairness problem:

• Construct-irrelevant difference: the item is picking up something other than the trait it’s supposed to measure — unfamiliar vocabulary or cultural context, a translation artifact, a response format one group has less experience with, item content tied to a life experience unevenly distributed across groups. This is the case DIF analysis exists to surface.
• Construct-relevant difference: the item is legitimately harder or easier for one group because the groups genuinely differ on some component the construct is supposed to capture, and removing the item would narrow what the scale measures rather than fix a bias. Flagging DIF doesn’t resolve which explanation applies — that requires substantive content review, ideally by people with expertise in both the measured construct and the groups being compared.

Once an item is flagged and substantively reviewed, the usual options are: retain it with documented justification (if the difference is judged construct-relevant), revise the wording or format and re-test, or drop it from the scale. Because MH needs adequate cell counts in every stratum to produce a stable pooled estimate, DIF analyses are also sensitive to sample size — a small pilot sample, especially with a rare or hard item concentrating responses into a few extreme strata, often can’t support a reliable MH estimate at all, independent of whether real DIF is present.

Beyond Mantel-Haenszel: Other DIF Detection Methods

MH is the standard starting point, but it isn’t the only method, and each alternative trades away some of MH’s simplicity for sensitivity to patterns MH can miss:

• Logistic regression DIF (Swaminathan & Rogers, 1990) regresses the item response on the matching score, group, and a group×score interaction term. The main-effect coefficient on group captures uniform DIF and the interaction term captures non-uniform DIF in the same model, which the basic MH statistic can’t do on its own.
• IRT-based methods — Lord’s chi-square test on the difference between groups’ estimated item parameters, or a likelihood-ratio test comparing a model that constrains an item’s parameters equal across groups against one that frees them — work directly on the calibrated item characteristic curves rather than on raw score strata, at the cost of needing a fitted IRT model in the first place (see Item Response Theory for Research Scales and The Rasch Model).
• Raju’s area measures quantify DIF as the signed or unsigned area between two groups’ item characteristic curves across the trait range, which is a natural fit for summarizing non-uniform patterns where a simple odds ratio understates the effect.
• For polytomous items (Likert-type scales rather than right/wrong items), MH generalizes to a Mantel-form test on ordered categories, and SIBTEST-family methods extend the same matched-groups logic to bundles of items rather than one at a time.

Frequently Asked Questions

Is DIF the same thing as bias?

No. DIF is a statistical pattern — a necessary condition for concluding an item is biased, but not a sufficient one. An item can show real DIF for a construct-relevant reason (see above), and a biased item could theoretically produce weak DIF if its distorting effect happens to be uniform and small relative to sampling noise. DIF flags candidates for substantive review; it doesn’t replace that review.

What’s the practical difference between DIF and impact?

Impact is unconditional: it’s just the raw score or pass-rate gap between groups. DIF is conditional: it’s the gap that remains after matching people on the trait itself. A test can show large impact with no DIF anywhere in it (the groups genuinely differ on the trait), or show DIF on several items that roughly cancel out at the total-score level, producing little visible impact even though individual items are behaving unevenly.

How large do the reference and focal groups need to be?

There’s no single fixed minimum, but the practical constraint is cell counts: every stratum needs enough respondents in both groups for the pooled odds ratio and its variance to be stable, not just the overall sample. Published applications commonly work with groups in the low hundreds to several thousand per group; very small pilot samples, especially on a short anchor set that produces few usable strata, typically can’t support a reliable MH estimate regardless of whether true DIF exists.

Can DIF favor the reference group instead of the focal group?

Yes — DIF is symmetric. “Reference” and “focal” are just labels assigned before the analysis (often majority/minority group by convention, or any two groups being compared), and either can come out favored on any given item. The sign convention on MH D-DIF (negative favors the reference group, positive favors the focal group) exists precisely because both directions are common in practice.

Related CASRAI Resources

• Psychometrics: How Researchers Measure Things You Can’t Observe — the broader measurement framework DIF analysis sits inside.
• Item Response Theory for Research Scales — the IRT parameters (difficulty, discrimination) that uniform and non-uniform DIF map onto.
• The Rasch Model: Item-Invariant Measurement and the Person-Item Map — the one-parameter IRT model whose specific-objectivity property DIF violates when it’s present.
• Construct Validity — evidence types and threats, for judging whether a flagged item’s difference is construct-relevant or not.
• Reliability in Research Measurement — classical test theory concepts that DIF analysis is often run alongside.
• Content Validity — expert-panel review methods relevant to substantively interpreting a flagged item.
• Cronbach’s Alpha — the internal-consistency statistic most often reported alongside DIF results.
• Convergent and Discriminant Validity — another evidence type for what a scale is actually measuring.
• Research Methods hub — the full cluster hub for study design, sampling, analysis, and measurement content.

The Mantel-Haenszel procedure used above is the same stratified 2×2-table logic used more generally for pooling stratified epidemiological and observational data (a common odds ratio across confounder strata) — DIF is one specific application of that general method to item-level test data.