Inter-Rater Reliability: Choosing the Right Coefficient

On this page: a selection table that maps your data type and rater design to the correct agreement statistic; what each coefficient actually assumes; a fully worked 2×2 showing how Cohen’s kappa collapses from 0.90 to −0.02 across three tables that all have exactly 95% agreement and exactly five disagreements; why that paradox is the reason Gwet’s AC1 exists; and why the Landis & Koch verbal labels are a 1977 convention rather than a pass/fail scale.

“Inter-rater reliability” names a question, not a statistic. The question is whether two or more raters, coders, observers or instruments reach the same result on the same material. At least six different coefficients answer it, they are not interchangeable, and the one you pick is determined by your data and your design — not by which one your software puts first in the menu. Most guides explain each coefficient in isolation and leave the selection to the reader. Selection is the part reviewers actually challenge, so it is the part this page starts with.

The selection table

Work down the first two columns — what the raters produced, and how the raters were assigned — and read off the statistic.

Data the raters produced	Rater design	Use	Why
Nominal categories (diagnosis, include/exclude, theme label)	Exactly 2 raters, both rating every case	Cohen’s kappa (κ)	The original chance-corrected agreement statistic; defined only for two raters.
Ordered categories (severity 0–4, Likert, tumour grade)	Exactly 2 raters, both rating every case	Weighted kappa (linear or quadratic)	Unweighted kappa treats a 1-vs-4 disagreement as identical to a 1-vs-2 disagreement. Weighting recovers the ordering.
Nominal categories	3+ raters, and not necessarily the same raters for every case	Fleiss’ kappa	Generalises chance-corrected agreement to any fixed number of raters per subject, drawn from a larger pool.
Nominal or ordinal, with skewed prevalence (one category dominates)	2 or more raters	Gwet’s AC1/AC2, reported alongside kappa and raw agreement	Built specifically so the chance-agreement term does not explode when one category is rare. See the paradox below.
Any level of measurement — nominal, ordinal, interval or ratio	Any number of raters, missing ratings allowed	Krippendorff’s alpha (α)	The only coefficient in common use that handles incomplete rating matrices and varying rater counts natively, with one benchmark scale across all data types.
Continuous / interval scores (range of motion, VAS pain, a summed scale score)	2 or more raters	ICC — and you must then choose which of the six forms	Variance-decomposition reliability for continuous data. The form choice is a second, independent decision.
Anything	Anything	Raw percent agreement — always report it as well, never instead	Uninterpretable alone because it does not correct for chance, but it is the only number that makes a paradoxical kappa diagnosable by a reader.

Two entries in that table are decision points rather than answers, and both are covered below: the ICC row hands off to a further six-way choice, and the Gwet’s AC1 row only makes sense once you have seen the paradox it was designed for.

What is not an inter-rater reliability statistic

• Pearson’s r. Two raters who disagree on every single case — one scoring exactly five points higher than the other, every time — produce r = 1.0. Correlation measures covariation, not agreement. It is structurally blind to systematic offset, which is precisely the failure mode a reliability study exists to detect. See CASRAI’s guide to the correlation coefficient for what r does and does not license.
• Cronbach’s alpha. This measures internal consistency across the items of a scale within a single administration — whether the items hang together — not whether raters agree. It is a different question with a confusingly similar name; see Cronbach’s alpha.
• A chi-square test of association. Significant association between two raters’ judgments is compatible with severe, systematic disagreement. Association is not agreement.

Step 1 — classify the data, not the topic

The first branch is the level of measurement of what the raters wrote down, which is often not the level of measurement of the underlying construct.

• Nominal — unordered labels. Two raters coding an interview transcript into themes, or screening abstracts include/exclude. Kappa family.
• Ordinal — ordered categories where the distance between adjacent levels is not assumed equal. Weighted kappa, or Krippendorff’s alpha with ordinal difference weights.
• Interval or ratio — genuine numeric scores. ICC.

The routine error here runs in one direction: applying unweighted Cohen’s kappa to an ordered scale because the ratings happen to be recorded as categories. If a rater scoring 4 when the other scored 3 is a smaller error than scoring 4 when the other scored 0, unweighted kappa is throwing that information away and will understate agreement.

Step 2 — classify the rater design

Three design facts determine the rest of the choice.

1. How many raters? Cohen’s kappa is defined for exactly two. Three or more requires Fleiss’ kappa, Krippendorff’s alpha, or an ICC.
2. Are they the same raters for every case? If a fixed pair rates all 200 transcripts, that is a crossed design. If each transcript is rated by whichever two of eight coders were available, raters are nested within cases, and Fleiss’ kappa or Krippendorff’s alpha is the honest choice — Cohen’s kappa has no way to represent it.
3. Is the rating matrix complete? Real coding projects lose cells: a coder drops out, a case is unrateable, a third opinion was sought only for hard cases. Cohen’s and Fleiss’ kappa require complete data and force listwise deletion. Krippendorff’s alpha does not — it is defined over whatever ratings exist. In a project with meaningful missingness this alone can decide the choice.

A fourth question governs interpretation rather than selection: are these specific raters the only ones of interest, or a sample standing in for a wider pool of similarly trained raters you intend to generalise to? That distinction has no effect on kappa but changes which ICC form is correct, and it changes what a good result entitles you to claim.

The coefficients

Percent agreement

The proportion of cases the raters scored identically. Report it always; interpret it never on its own. Its defect is the reason every other statistic on this page exists: if 95% of the abstracts you are screening are excludes, two raters who both simply say “exclude” to everything achieve 95% agreement while demonstrating nothing at all. Its virtue is that it is the only figure that lets a reader diagnose a paradoxical kappa. A paper reporting κ = 0.26 without the raw agreement figure is uninterpretable in both directions.

Cohen’s kappa

Introduced by Jacob Cohen in 1960 (“A Coefficient of Agreement for Nominal Scales”, Educational and Psychological Measurement 20(1), 37–46), kappa asks how much of the observed agreement exceeds what the raters would have achieved by guessing at their own observed base rates:

κ = (P_o − P_e) / (1 − P_e)

where P_o is the observed proportion of agreement and P_e is the proportion expected by chance, computed from the product of the two raters’ marginal totals. κ = 1 is perfect agreement, κ = 0 is agreement no better than chance, and negative values indicate agreement worse than chance — which happens, and is usually a sign of a coding-scheme or data-orientation problem rather than genuine perversity.

Its two hard limits: exactly two raters, and unordered categories. Its soft limit — the behaviour of P_e under skewed marginals — is the subject of the next section and matters more in practice than either hard limit.

Weighted kappa

Cohen’s 1968 extension for ordered categories. Instead of scoring every disagreement as a total miss, each off-diagonal cell carries a weight reflecting how far apart the two ratings were. Two weighting schemes dominate:

• Linear weights — the penalty grows in proportion to the number of categories separating the ratings.
• Quadratic weights — the penalty grows with the square of that distance, so large disagreements are punished disproportionately.

The choice is substantive, not cosmetic: quadratic weighting is more forgiving of adjacent-category disagreement and will generally return a higher number than linear weighting on the same table. Decide and declare it before you look at the results, and state which scheme you used — “weighted kappa = 0.71” without the weighting scheme is as incomplete as “ICC = 0.82” without the form.

One connection worth knowing: quadratically weighted kappa on an ordered scale is, under standard conditions, numerically equivalent to a two-way ICC computed on the same data treated as numeric. That is not a coincidence to exploit but a consistency check — if your quadratic weighted kappa and your ICC disagree sharply, one of them is computing something other than what you think.

Fleiss’ kappa

Fleiss (1971) generalised chance-corrected agreement to a fixed number of raters per subject where the raters need not be the same people across subjects. This is the right statistic for a coding project staffed from a pool. Note what it is not: it is not “Cohen’s kappa for three raters” in the sense of a crossed design with three fixed raters, and it does not reduce to Cohen’s kappa in the two-rater case (it reduces to Scott’s pi, which computes P_e from pooled rather than individual marginals). If you have three fixed raters and want pairwise detail, reporting the three pairwise Cohen’s kappas alongside an overall Fleiss’ kappa is more informative than either alone, because it exposes a single divergent rater that an aggregate figure hides.

Krippendorff’s alpha

The most general coefficient in routine use, and the standard in content analysis and computational linguistics. Alpha is built from observed and expected disagreement rather than agreement, which is what lets a single formula accept nominal, ordinal, interval and ratio data by swapping the difference function — and, critically, lets it accept any number of raters and an incomplete rating matrix without deleting cases.

Its conventional benchmarks come from Krippendorff himself and are stricter than the kappa conventions: rely on data at α ≥ 0.800; draw only tentative conclusions in the 0.667–0.800 band; discard below 0.667. Those are still conventions rather than derived thresholds, but they are at least stated by the coefficient’s own author with an explicit rationale about what the data will be used for. If you are coding text or media, see CASRAI’s guides to content analysis and coding qualitative interview data for how alpha fits into the wider coding workflow.

Gwet’s AC1

Gwet’s first-order agreement coefficient keeps kappa’s (P_o − P_e) / (1 − P_e) shape but redefines P_e. Kappa’s chance term assumes raters guess independently at their observed marginal rates, which makes P_e approach 1 as one category comes to dominate. Gwet’s chance term instead reflects the premise that chance agreement arises only on the subset of cases a rater is genuinely uncertain about, so it does not blow up under skew. For two raters and two categories it reduces to P_e = 2π(1 − π), where π is the mean prevalence of the category across raters. AC2 is the weighted variant for ordinal data.

AC1 is not a strictly better kappa, and it should not be presented as one — it answers a slightly different question about what “chance” means, and there is genuine methodological disagreement about whether its premise is the right one. The defensible use is to report it alongside kappa and raw agreement when prevalence is skewed, and to say plainly why. Reporting AC1 alone, after seeing that kappa was disappointing, is coefficient shopping.

ICC — and which of the six forms

For continuous ratings the intraclass correlation coefficient is the standard, but “the ICC” is a family, not a statistic. Three independent decisions define which member you are computing, and getting one wrong means the number you report answers a different question than your design asked:

• Model — one-way random (each subject rated by a different random set of raters), two-way random (the same raters rate everyone, and they represent a wider pool), or two-way mixed (the same raters rate everyone, and only those raters matter).
• Type — single measures, if one rater’s single score will be used in practice; average measures (the “k” forms), if a multi-rater mean will be used. Averaging mechanically raises reliability, so the k forms are always the higher number and are only honest when a mean is genuinely what gets used.
• Definition — absolute agreement, which counts a systematic offset between raters against you, or consistency, which does not. Absolute agreement is almost always the defensible choice when different raters’ scores will be used interchangeably.

The two notations you will meet are Shrout & Fleiss (1979), which writes ICC(2,1) and ICC(3,k), and McGraw & Wong (1996), which writes ICC(A,1) and ICC(C,k) so that the agreement-versus-consistency choice is visible in the label itself. Koo & Li’s 2016 guideline in the Journal of Chiropractic Medicine is the most-cited practical selection aid, and its interpretive bands — below 0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, above 0.90 excellent — are conventions in the same sense as Landis & Koch’s, with the same caveats. Koo & Li’s more useful recommendation is the one most often skipped: judge reliability from the 95% confidence interval, not the point estimate.

CASRAI’s guide to the intraclass correlation coefficient works through all six forms, the decision table, sample size and reporting in full. This page deliberately stops at “you need an ICC, and here is which decisions define it” rather than repeating that treatment.

The kappa paradox, worked

This is the single most common misinterpretation in the literature, and it is best seen arithmetically rather than described. Below are three 2×2 tables from a hypothetical screening task — two raters, 100 abstracts, include or exclude. All three have exactly 95% observed agreement and exactly five disagreements. Only the prevalence of the “include” category differs.

Table	Both include	A include / B exclude	A exclude / B include	Both exclude	Po	Pe	Cohen’s κ	Gwet’s AC1
A — balanced (~50% include)	48	3	2	47	0.95	0.500	0.90	0.90
B — skewed (~4% include)	1	3	2	94	0.95	0.932	0.26	0.95
C — extreme (~3% include, no agreed includes)	0	2	3	95	0.95	0.951	−0.02	0.95

Table A yields κ = 0.90, which the conventional labels call “almost perfect”. Table B yields κ = 0.26 — “fair”. Table C yields a negative kappa, nominally “worse than chance”, from raters who agreed on 95 of 100 abstracts. The raters did not get worse. The data did not get noisier. The number of disagreements is identical in all three.

The mechanism is entirely in P_e. In Table B, rater A called 4 abstracts “include” and rater B called 3; kappa computes expected chance agreement as (0.04 × 0.03) + (0.96 × 0.97) = 0.932. Kappa then asks how far 0.95 exceeds 0.932 relative to the 0.068 of headroom remaining — and 0.018 out of 0.068 is 0.26. When one category dominates, chance agreement is already nearly perfect, so almost no room is left for a rater to demonstrate skill. Kappa is not malfunctioning; it is answering its own question correctly, and its question turns out to be a strange one to ask about a rare-event task.

This is the first of the two paradoxes documented by Feinstein and Cicchetti (1990). The second is the mirror image: unbalanced marginals, where two raters who disagree about the base rate can produce a higher kappa than two raters who agree about it, given the same observed agreement.

Gwet’s AC1, in the final column, returns 0.90, 0.95 and 0.95 — stable across all three tables, because its chance term does not scale with prevalence. That stability is the entire reason AC1 exists, and Table C is what its inventor was arguing about. A simpler and older correction, PABAK (prevalence-adjusted bias-adjusted kappa, Byrt and colleagues 1993), is just 2P_o − 1 and returns 0.90 for all three tables; it is transparent but discards the information about prevalence entirely, which is a real cost, not a free fix.

What to do when you hit it

1. Report P_o, the full 2×2 or n×n table, and the marginal prevalences alongside kappa. This is non-negotiable and costs nothing. A reader cannot detect the paradox from a kappa alone.
2. Report a prevalence-robust coefficient alongside — AC1, or PABAK with its limitation stated — and say in the methods that you pre-specified this because the category was expected to be rare.
3. Do not quietly swap coefficients after seeing the result. The distinction between “we pre-specified AC1 because prevalence was expected to be under 10%” and “kappa was 0.26 so we report AC1 = 0.95” is the whole of research integrity in one sentence.
4. Consider whether the rare category is the thing you actually care about. For a rare-event screening task, positive and negative specific agreement — agreement computed separately within each category — is often far more informative than any single overall coefficient.

The Landis & Koch labels are a convention, not a scale

Nearly every paper reporting a kappa attaches a verbal label from Landis and Koch’s 1977 Biometrics paper “The Measurement of Observer Agreement for Categorical Data”: below 0.00 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.

Those bands are not derived from anything. They were not obtained from a distribution, a decision-theoretic argument, or an error-rate calculation. Landis and Koch proposed them as a convenience and said as much — the paper describes the divisions as clearly arbitrary while offering them as useful benchmarks. That is a much weaker claim than the way they are routinely cited, and later methodologists have been blunter still: Ludbrook (2002) argued the approach has no sound theoretical basis and can positively mislead investigators.

The practical consequences:

• “Substantial agreement” is not a finding. It is a label applied to a number. Write “κ = 0.68 (95% CI 0.55–0.81)”, not “agreement was substantial”.
• There is no universal threshold at which a measure becomes fit for use. The bar depends on what the rating decides. A coding scheme for exploratory qualitative themes and a rating that determines whether a patient receives a treatment do not warrant the same threshold, and no 1977 convention can tell you what either bar should be.
• Report the confidence interval, and judge from it. A kappa of 0.61 with a CI running from 0.38 to 0.84 straddles three of the six bands. Reporting only the point estimate and its label conveys a precision the data do not support — the same objection Koo & Li raise about ICC bands.
• Set your threshold in the protocol, with a justification, before you compute anything. Then a disappointing result is a finding rather than a negotiation.

What to report

A complete inter-rater reliability report states, at minimum:

• The coefficient by full name, including any variant — “quadratic weighted kappa”, not “weighted kappa”; the ICC model, type and definition, not “ICC”.
• The point estimate and its 95% confidence interval.
• Raw percent agreement and the marginal distribution of categories, so a reader can detect a prevalence problem.
• The number of raters, whether they were the same across all cases, and how much of the rating matrix was complete.
• Whether raters were independent and blinded to each other’s ratings, and what training or calibration preceded coding.
• The software and version, since defaults differ: R’s irr and psych, Python’s pingouin, SPSS’s Reliability Analysis and Stata’s kappa/kap do not all default to the same form or the same CI method.
• The threshold you pre-specified and the reasoning behind it.

The Guidelines for Reporting Reliability and Agreement Studies (GRRAS), published by Kottner and colleagues in the Journal of Clinical Epidemiology in 2011, is the established reporting checklist for this class of study; COSMIN provides the broader framework for evaluating measurement instruments.

A note on what a good coefficient does and does not buy you

High inter-rater reliability means raters apply your coding scheme consistently. It does not mean the scheme measures what you intended, and it does not mean the ratings are accurate — two raters trained on the same wrong definition will agree beautifully. Reliability is a precondition for validity, never evidence of it. And a reliability coefficient does not by itself tell a clinician how much change in an individual score exceeds measurement noise; converting reliability into a standard error of measurement and then a minimal detectable change threshold is a separate step.

For the wider framing — how inter-rater reliability sits alongside test-retest, internal consistency and parallel-forms reliability — see CASRAI’s guide to reliability in research measurement and the comparison of test-retest vs. inter-rater reliability, which sets out how the two isolate different sources of error. For the measurement-theory background, see psychometrics. This guide sits in CASRAI’s research methods cluster.

Frequently asked questions

What is a good inter-rater reliability score?

There is no universal answer, and any source that gives one is quoting a convention. The Landis & Koch bands (0.61–0.80 “substantial”, above 0.81 “almost perfect”) are the usual reference for kappa and Krippendorff recommends α ≥ 0.800 for his alpha, but both are conventions rather than derived thresholds. The defensible practice is to pre-specify a threshold justified by what the rating will be used for, and to judge the result from its confidence interval.

Why is my kappa low when my raters agree on 95% of cases?

Almost certainly the prevalence paradox. When one category dominates, the chance-agreement term kappa subtracts becomes very large, leaving almost no headroom above chance for the raters to occupy. See the worked tables above, where 95% agreement produces kappas of 0.90, 0.26 and −0.02 depending only on prevalence. Report raw agreement and the marginal distribution alongside kappa, and consider a prevalence-robust coefficient such as Gwet’s AC1.

Cohen’s kappa or Fleiss’ kappa?

Cohen’s kappa is defined for exactly two raters. Fleiss’ kappa handles three or more, and does not require the same raters to rate every subject. With three fixed raters, reporting the pairwise Cohen’s kappas as well as an overall Fleiss’ kappa is more diagnostic, because it reveals a single divergent rater that an aggregate figure conceals.

When should I use Krippendorff’s alpha instead of kappa?

When your rating matrix is incomplete, when the number of raters varies across cases, when your data are ordinal or interval rather than nominal, or when you want one coefficient applied consistently across several variables measured at different levels. It is the default in content analysis for exactly these reasons.

Can I use ICC for categorical data, or kappa for continuous scores?

No in both directions — each is a category error. ICC decomposes variance and presumes numeric scores; kappa counts exact category matches and presumes discrete labels. The one legitimate bridge is that quadratically weighted kappa on an ordered scale is, under standard conditions, numerically equivalent to a two-way ICC on the same data.

Is Gwet’s AC1 better than Cohen’s kappa?

It is not a strict improvement; it defines chance agreement differently, and whether that definition is the right one is genuinely contested in the methodological literature. Its advantage is stability under skewed prevalence, where kappa behaves paradoxically. Use it alongside kappa and raw agreement, pre-specified in your protocol — not as a substitute chosen after kappa disappointed you.

Do I need to correct for chance at all?

Yes, for categorical ratings. Raw agreement rewards raters for guessing the dominant category and cannot distinguish skill from base rates. The reasonable position is not to abandon chance correction but to report the uncorrected figure and the marginals alongside it, so a reader can see what the correction is doing.

How many cases should be double-rated?

There is no single rule, and the answer depends on the width of confidence interval you can accept, the number of categories, and the expected prevalence of the rarest one — rare categories need substantially more cases before an estimate stabilises. Double-rating a random subsample is standard where full double-rating is infeasible, but the subsample must be random and the proportion must be reported. A formal sample-size calculation for a reliability design is preferable to a rule of thumb; a power calculation built for a group-comparison test does not transfer.

What sources should I cite for these coefficients?

Cohen (1960) for kappa; Cohen (1968) for weighted kappa; Fleiss (1971) for the multi-rater generalisation; Landis & Koch (1977) for the verbal benchmarks, cited as the convention they are; Shrout & Fleiss (1979) and McGraw & Wong (1996) for the ICC forms and notation; Koo & Li (2016) for practical ICC selection and reporting; Feinstein & Cicchetti (1990) for the paradoxes; Byrt and colleagues (1993) for PABAK; and Krippendorff’s Content Analysis for alpha and its benchmarks.