On this page: what the intraclass correlation coefficient (ICC) measures, why the specific form you choose (ICC(1,1), ICC(2,1), ICC(3,1), and their average-measures “k” counterparts) is the single most common source of error in published reliability research, how to interpret the result, how ICC compares to Cohen’s/Fleiss’ kappa, Pearson’s r, Bland-Altman analysis, and Cronbach’s alpha, sample-size guidance for reliability studies, how to report results, and how to compute ICC in R, Python, and SPSS.
What ICC Measures
The intraclass correlation coefficient (ICC) is a reliability statistic for continuous measurements. It expresses the proportion of the total variance in a set of scores that is due to true differences between subjects, as opposed to variance introduced by measurement error, rater disagreement, or occasion-to-occasion inconsistency:
ICC = variance(between subjects) / [variance(between subjects) + variance(error)]
An ICC close to 1.0 means that most of the variability in the data reflects genuine differences between the people or specimens being measured, and very little reflects noise from the measurement process itself. An ICC close to 0 means the opposite: raters, instruments, or occasions disagree with each other by roughly as much as subjects differ from one another, so the measurement is not distinguishing subjects reliably.
ICC is used for two related but distinct purposes, both of which require continuous or ordinal-treated-as-continuous data:
- Inter-rater reliability — do two or more raters/observers/instruments produce consistent scores for the same subjects?
- Test-retest reliability — does the same rater/instrument produce consistent scores for the same subjects across repeated occasions?
See CASRAI’s comparison of test-retest vs. inter-rater reliability for how those two reliability questions differ conceptually — ICC is the statistic commonly used to quantify either one when the underlying data are continuous.
A note on the search term: ICC is very frequently typed and searched as “inter class correlation” or “interclass correlation.” This is a common misspelling of intraclass correlation — the “intra” (within) is doing real conceptual work here, since the statistic is built from a within-subject/between-subject variance decomposition, not a correlation “between classes” of anything. If you found this page searching either term, you’re in the right place.
Why the Specific ICC “Form” Is the Single Biggest Source of Error
Unlike Pearson’s r, “the ICC” is not one statistic — it is a family of at least six (and, depending on convention, up to ten) distinct coefficients that can produce meaningfully different numbers from the exact same dataset. Reporting “the ICC was 0.82” without specifying which form was used is functionally incomplete, and choosing the wrong form for a given study design is the most frequently cited methodological error in published reliability research. The two conventions researchers cite are:
- Shrout & Fleiss (1979) — the original notation, ICC(model, type), e.g. ICC(2,1), ICC(3,k).
- McGraw & Wong (1996) — a later, more explicit notation, e.g. ICC(A,1) for absolute agreement or ICC(C,k) for consistency, that makes the agreement-vs-consistency choice part of the label itself rather than leaving it implicit in the model.
Three independent decisions combine to define a specific ICC form. Get any one wrong and the number you report answers a different question than the one your study design actually asked.
Decision 1 — Model: One-Way or Two-Way?
- One-way random effects (Shrout & Fleiss “Case 1”) — each subject is rated by a different, randomly selected set of raters. There is no way to separate rater variance from residual error, because raters aren’t crossed with subjects. Produces ICC(1,1) or ICC(1,k).
- Two-way random effects (Case 2) — the same set of raters rates every subject, and those raters are themselves treated as a random sample from a larger population of possible raters you want to generalize beyond. Produces ICC(2,1) or ICC(2,k).
- Two-way mixed effects (Case 3) — the same set of raters rates every subject, but those specific raters are the only ones of interest — you are not trying to generalize to other, hypothetical raters. Produces ICC(3,1) or ICC(3,k).
The practical test: if your raters are a fixed panel you’re stuck with for this study (the two study clinicians, this one instrument), use the two-way mixed model (Case 3). If your raters are meant to represent a broader pool of similarly-trained raters you want your conclusion to generalize to, use the two-way random model (Case 2).
Decision 2 — Type: Single Measures or Average Measures?
This reflects what will actually be used in practice. If a single rater’s single score will be used going forward (a clinician’s one assessment, one lab technician’s one reading), report the single-measures ICC (the “,1” forms: ICC(2,1), ICC(3,1)). If scores from multiple raters will be averaged before use (a consensus score from a 3-rater panel), report the average-measures ICC (the “,k” forms: ICC(2,k), ICC(3,k)) — averaging multiple raters’ scores mechanically increases reliability, so the “k” forms are always numerically higher than their single-measures counterparts for the same data, and the two answer genuinely different practical questions.
Decision 3 — Definition: Consistency or Absolute Agreement?
- Absolute agreement asks: do raters assign the same numeric value? A systematic offset between raters (one rater consistently scoring two points higher than another) counts against reliability.
- Consistency asks: do raters rank subjects the same way, even if there’s a systematic offset between them? A rater who is consistently two points higher than another, but tracks their relative ordering of subjects perfectly, still scores well on the consistency definition.
Absolute agreement is almost always the more defensible choice when scores from different raters or occasions will be used interchangeably in practice (e.g., a clinical score that different clinicians will read at different visits) — a systematic offset is a real problem in that use case. Consistency is defensible when only relative ranking matters, or when a known, correctable offset (e.g., a calibration constant between two instruments) is not itself the thing being tested.
Decision Table: Which Form for Which Design
| Your design | Model | Type | Definition | Form (Shrout & Fleiss) |
|---|---|---|---|---|
| Each subject rated by a different random rater; single rating used | One-way random | Single | Absolute agreement (only option under this model) | ICC(1,1) |
| Same raters rate all subjects; raters are a random sample you generalize beyond; single rating used, offsets matter | Two-way random | Single | Absolute agreement | ICC(2,1) |
| Same raters rate all subjects; those specific raters are the only ones of interest; single rating used, only ranking matters | Two-way mixed | Single | Consistency | ICC(3,1) |
| Same raters rate all subjects; those specific raters are the only ones of interest; single rating used, offsets matter | Two-way mixed | Single | Absolute agreement | ICC(A,1) in McGraw & Wong notation |
| Any of the above two-way designs, but a k-rater average (not a single rating) will be used going forward | Two-way random or mixed, as above | Average (k) | As above | ICC(2,k) or ICC(3,k) |
Because Koo & Li’s widely cited guideline collapses the two-way mixed model to “consistency” and the two-way random model to “absolute agreement” by default, most statistical packages (SPSS’s Reliability Analysis, R’s irr::icc()) default the same way — but McGraw & Wong’s fuller ten-form framework confirms both the random and mixed models can, in principle, be paired with either definition. When your software’s defaults don’t match your actual design decision, override them explicitly and say so in your methods section.
Interpreting ICC Values
The most widely cited interpretation benchmarks come from Koo & Li’s 2016 guideline in the Journal of Chiropractic Medicine:
- Less than 0.50 — poor reliability
- 0.50 to 0.75 — moderate reliability
- 0.75 to 0.90 — good reliability
- Greater than 0.90 — excellent reliability
Two caveats matter more than the thresholds themselves. First, these are conventions adopted for convenience, not statistical laws — a different field or a higher-stakes application (e.g., a diagnostic threshold that determines treatment) may reasonably demand a stricter bar than a general convention supplies. Second, and more important: Koo & Li’s own guideline recommends basing the reliability judgment on the 95% confidence interval of the ICC estimate, not the point estimate alone. A point estimate of 0.82 sounds solidly “good,” but if that estimate came from a small study and its 95% CI actually runs from 0.55 to 0.95, the honest read spans moderate through excellent — reporting only “0.82, good reliability” overstates the precision the data actually support.
ICC vs. Alternatives: Kappa, Pearson r, Bland-Altman, and Cronbach’s Alpha
ICC is frequently confused with, or substituted for, several other statistics that answer related but genuinely different questions:
- Cohen’s kappa / Fleiss’ kappa — the categorical-data equivalents of ICC. Use kappa when raters assign subjects to discrete categories (a diagnosis, a pass/fail rating); use ICC when raters assign a continuous or ordinal score. Applying ICC to nominal categories, or kappa to continuous scores, is a category error.
- Pearson’s correlation coefficient (r) — r is not a reliability or agreement statistic, even though it is routinely misused as one. Two raters whose scores are perfectly correlated (r = 1.0) but who are systematically offset by a constant (one always scoring five points higher) would show perfect Pearson correlation while genuinely disagreeing on every subject’s actual value. ICC’s absolute-agreement forms are specifically built to catch that kind of systematic bias; r structurally cannot.
- Bland-Altman analysis — used to assess agreement between two measurement methods (e.g., a new device against a reference standard), producing a bias estimate and limits of agreement rather than a single coefficient. Bland-Altman and ICC are complementary, not competing: ICC gives a single reliability number and confidence interval, Bland-Altman visualizes where and how two methods disagree across the measurement range. Many methodologists recommend reporting both for a method-comparison study.
- Cronbach’s alpha — measures internal consistency among multiple items on a scale (do the items on a questionnaire measure the same underlying construct?), not agreement between raters or occasions. See CASRAI’s guide to Cronbach’s alpha for the full treatment, including a caveat worth flagging here: under certain conditions (a two-way random model, average-measures type), Cronbach’s alpha and ICC(2,k)/ICC(3,k) are mathematically identical. This is a genuine overlap, not a coincidence: alpha can be understood as a special case of ICC applied to items-as-raters rather than raters-as-raters.
All four sit within the broader question of whether an instrument measures its intended construct consistently and correctly — see CASRAI’s guide to types of validity in research for how reliability (addressed by ICC, kappa, and alpha) relates to, but is distinct from, validity.
Sample Size for Reliability Studies
ICC-based reliability studies need adequate sample size on two separate dimensions, and underpowering either one produces an unstable estimate even if the other looks generous:
- Number of subjects — too few subjects, and the between-subject variance term in the ICC formula is itself poorly estimated, producing wide, unstable confidence intervals regardless of how many raters you use. Formal sample-size calculations for reliability studies (based on the expected ICC, the number of raters, and the desired CI width) are available in dedicated reliability-study sample-size methods and software (e.g., R’s
ICC.Sample.Size-style approaches) — a generic post-hoc power analysis built for a group-comparison test does not directly transfer to a reliability design and should not be substituted for one. - Number of raters (k) — more raters generally narrows the CI and, for average-measures forms, mechanically increases the ICC itself, but only if the additional raters are genuinely representative of the rater population you intend to generalize to (for the two-way random model) rather than added simply to inflate the number.
As a general rule of thumb reported across reliability-methods literature, studies with fewer than 30 subjects tend to produce confidence intervals too wide to support firm conclusions about a specific reliability tier (poor vs. moderate vs. good) — treat this as a floor to plan above, not a target to plan to exactly, and prefer a formal calculation over a rule of thumb where feasible.
How to Report ICC Results
The Guidelines for Reporting Reliability and Agreement Studies (GRRAS), published by Kottner and colleagues in the Journal of Clinical Epidemiology (2011), is the most widely cited reporting checklist specific to reliability and agreement studies, alongside the general COSMIN initiative’s guidance on evaluating outcome measurement instruments. Applied specifically to ICC, Koo & Li’s guideline recommends every report state, at minimum:
- The model used (one-way random, two-way random, or two-way mixed).
- The type (single measures or average of k measures), and the value of k if applicable.
- The definition (consistency or absolute agreement).
- The ICC point estimate and its 95% confidence interval, not the point estimate alone.
- The software (and version) used to compute it, since default forms and CI-computation methods differ across packages.
A minimally complete reporting sentence looks like: “Inter-rater reliability was assessed using a two-way random-effects model with absolute agreement, single-rater type — ICC(2,1) = 0.84, 95% CI [0.71, 0.92] — calculated in R (irr package, v0.84.1).”
Calculating ICC: R, Python, and SPSS
- R — two commonly used packages:
irr::icc(), which most closely follows Shrout & Fleiss notation and lets you specifymodel(“oneway”/”twoway”),type(“consistency”/”agreement”), andunit(“single”/”average”) directly as arguments; andpsych::ICC(), which returns all six Shrout & Fleiss forms in one call so you can compare them side by side. - Python — the
pingouinlibrary’sintraclass_corr()function returns all ten McGraw & Wong forms (ICC1 through ICC3k, plus their two-way variants) in a single results table, including F-statistics, p-values, and 95% confidence intervals for each form. - SPSS — Analyze → Scale → Reliability Analysis → Statistics → check “Intraclass correlation coefficient,” then choose the Model (One-Way Random, Two-Way Random, or Two-Way Mixed) and Type (Consistency or Absolute Agreement) from the dropdowns that appear.
Whichever tool you use, confirm the model/type/definition it actually computed matches the design decision in the decision table above before reporting the number — software defaults vary, and a default silently accepted is a common source of the form-selection error this page opened with.
Frequently Asked Questions
Is “inter-class correlation” the same as “intraclass correlation”?
Yes — “inter class correlation” and “interclass correlation” are common misspellings/mishearings of intraclass correlation coefficient. There is no separate statistic called “inter-class correlation” in standard use; if you encountered that phrase, it refers to ICC.
Can ICC be negative?
Yes. A negative ICC indicates that within-subject (error) variance exceeds between-subject variance — in practice, this usually signals a genuinely unreliable measurement, a coding/data-entry error, or a mismatch between the ICC form used and the actual study design, and is conventionally reported as 0 for interpretation purposes while investigating the underlying cause.
Do I need to run all six (or ten) ICC forms?
No — report the one form that matches your actual study design (see the decision table above). Running and reporting several forms and highlighting whichever looks best is a form of selective reporting; decide the model, type, and definition from your design before looking at the results.
What’s a good number of raters for an inter-rater ICC study?
There’s no universal minimum, but two raters is the practical floor for a two-way design, and reliability-methods literature generally favors three or more where feasible, since it both stabilizes the between-rater variance estimate and, for average-measures forms, more accurately reflects how scores will actually be used in a multi-rater practice setting.
See also: CASRAI’s guides to descriptive statistics for the variance and standard-deviation concepts ICC is built from, and questionnaire design for instrument-development context where ICC and Cronbach’s alpha are both routinely reported together.
