Written and maintained by CASRAI Editorial Board
Last updated
On this page: why Cohen’s kappa stops working once a third rater joins; the Fleiss’ kappa formula built from per-subject and per-category proportions; a fully worked example on a five-subject, four-rater rating matrix; why it is not simply “Cohen’s kappa for three or more raters”; and the Landis & Koch bands carried over from Cohen’s kappa, applied to a Fleiss’ kappa result.
Fleiss’ kappa (κ) is the chance-corrected agreement statistic for nominal categories when three or more raters score each subject, and the raters do not have to be the same people from one subject to the next — only the number of raters per subject has to stay fixed. That is the exact situation Cohen’s kappa cannot represent: Cohen’s kappa is defined for exactly two raters, full stop. This page assumes you already know you want Fleiss’ kappa rather than Cohen’s kappa, Krippendorff’s alpha or an ICC, and goes straight to the formula, a worked calculation, and how to interpret the result. For the full decision between coefficients — including exactly when Fleiss’ kappa is the right call — see CASRAI’s guide to choosing an inter-rater reliability coefficient; this page deliberately doesn’t repeat that selection logic.
Why Cohen’s kappa doesn’t extend to three or more raters
Cohen’s kappa’s expected-agreement term is built from two raters’ individual marginal distributions multiplied together. There is no natural way to extend that construction to three, five or twelve raters — whose marginals would you multiply, and in what order? Fleiss (1971) solved this by reformulating the problem: instead of tracking which specific rater said what, Fleiss’ kappa tracks, for each subject, how many of the n raters assigned each category, and compares the resulting agreement to what a shared pool of raters guessing at the observed overall category rates would produce by chance. That reformulation is also what lets the specific raters vary from subject to subject — a design a fixed two-rater or three-fixed-rater comparison cannot handle, and a common real situation for a coding project staffed from a rotating pool of coders or clinicians.
The formula
For N subjects, a fixed n raters per subject, and k categories, let xij be the number of raters who assigned subject i to category j.
Category proportion. The share of all N×n ratings that fell into category j:
pj = (1 / N n) × ∑i xij
Per-subject agreement. The proportion of rater pairs on subject i that agreed with each other:
Pi = (1 / n(n−1)) × ( ∑j xij² − n )
Observed agreement, averaged across all subjects:
P̄ = (1 / N) × ∑i Pi
Expected agreement, the chance level implied by how often each category was used overall:
P̄e = ∑j pj²
Fleiss’ kappa:
κ = (P̄ − P̄e) / (1 − P̄e)
The shape is identical to Cohen’s kappa — observed-minus-expected agreement, scaled by the headroom above chance — but P̄ and P̄e are built entirely from category counts, never from any individual rater’s identity. That is the mechanical reason Fleiss’ kappa needs only a fixed n per subject, not the same n people.
Worked example: five subjects, four raters, three categories
The illustration below is hypothetical — five case files, each independently scored by four raters into one of three categories (“A”, “B”, “C”) — chosen so every step is checkable by hand. It is not drawn from any real study.
| Subject | Category A | Category B | Category C |
|---|---|---|---|
| 1 | 4 | 0 | 0 |
| 2 | 3 | 1 | 0 |
| 3 | 2 | 2 | 0 |
| 4 | 1 | 0 | 3 |
| 5 | 0 | 4 | 0 |
Each row is xi,A, xi,B, xi,C for that subject; every row sums to n = 4, as it must, since every subject was rated by all four raters.
Step 1 — category proportions pj. Column totals: A = 4+3+2+1+0 = 10, B = 0+1+2+0+4 = 7, C = 0+0+0+3+0 = 3. Total ratings N×n = 5×4 = 20.
pA = 10/20 = 0.50 pB = 7/20 = 0.35 pC = 3/20 = 0.15
Step 2 — per-subject agreement Pi. With n = 4, the denominator n(n−1) = 12 for every subject:
Subject 1: (4² − 4) / 12 = 12/12 = 1.000
Subject 2: (3²+1² − 4) / 12 = 6/12 = 0.500
Subject 3: (2²+2² − 4) / 12 = 4/12 = 0.333
Subject 4: (1²+3² − 4) / 12 = 6/12 = 0.500
Subject 5: (4² − 4) / 12 = 12/12 = 1.000
Step 3 — observed agreement P̄. Average the five Pi values: (1.000+0.500+0.333+0.500+1.000)/5 = 3.333/5 = 0.667.
Step 4 — expected agreement P̄e. Sum the squared category proportions: 0.50² + 0.35² + 0.15² = 0.250 + 0.1225 + 0.0225 = 0.395.
Step 5 — kappa. κ = (0.667 − 0.395) / (1 − 0.395) = 0.272 / 0.605 ≈ 0.449.
Subjects 1 and 5 had unanimous raters (Pi = 1.000) while subject 3 was the most split; the overall figure sits well below either extreme once the chance correction for category C’s rarity is applied. That sensitivity to how the four Pi values are distributed, not just their average, is worth checking directly on real data — two studies with the same P̄ can carry a very different practical mix of near-unanimous and heavily-split subjects.
Not “Cohen’s kappa for three raters” — the two-rater case proves it
A natural but wrong intuition is that Fleiss’ kappa is simply Cohen’s kappa generalised to more raters, and should collapse back to Cohen’s kappa if you happen to only have two. It does not. Set n = 2 in the formula above and Fleiss’ kappa reduces to Scott’s pi, which computes its expected-agreement term from the two raters’ pooled marginal distribution (pj, combining both raters’ category counts together). Cohen’s kappa computes expected agreement from each rater’s individual marginal distribution multiplied together. Those are different quantities whenever the two raters use the categories at even slightly different rates, so the two coefficients can diverge on identical data. The practical consequence: if you have exactly three fixed raters who all rate every subject and you want to see whether one rater is the outlier, Fleiss’ kappa’s pooled-category construction hides that — it was built for a rotating pool, not for diagnosing a specific trio. Reporting the three pairwise Cohen’s kappas (A-B, A-C, B-C) alongside the overall Fleiss’ kappa is more diagnostic in that fixed-trio case, because it exposes a single divergent rater that the pooled figure conceals.
Reading the number: the Landis & Koch bands, carried over
Fleiss’ kappa is read on the same 0–1 chance-corrected scale as Cohen’s kappa, and the field has carried the same Landis & Koch (1977) verbal bands over to it rather than developing a separate convention:
| κ range | Label |
|---|---|
| < 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
The worked example above, κ ≈ 0.449, lands in the “moderate” band on this scale. Two things are worth keeping in mind before leaning on that label. First, Landis and Koch themselves describe these divisions as clearly arbitrary and offer them only as a convenient benchmark, not a validated or derived pass/fail scale — they were proposed for two-rater kappa and simply carried over by convention, without a separate derivation for the multi-rater case. Second, Fleiss’ kappa inherits the same prevalence paradox that affects Cohen’s kappa: when one category dominates the ratings, the expected-agreement term P̄e climbs toward 1, leaving little headroom for kappa to reward genuine agreement even when raters agree often in absolute terms. Report the raw percent agreement and each category’s overall prevalence alongside kappa, not the coefficient alone, so a reader can judge whether a low kappa reflects real rater disagreement or a skewed, headroom-starved category distribution.
The fixed-n, rotating-rater design
Fleiss’ kappa requires every subject to be rated by the same number of raters, but not the same people. This matters in practice for exactly the kind of project that motivates using the statistic in the first place: a large chart-review or content-coding effort staffed by a pool of coders, where scheduling means subject 1 might be rated by coders A, B and C while subject 40 is rated by coders D, E and F. Cohen’s kappa has no way to represent that design at all — it needs the same two raters throughout. Fleiss’ kappa handles it directly, provided the count of raters per subject (n) is constant. If n genuinely varies subject to subject — some cases got three raters, others five — Fleiss’ kappa is not the right tool either; Krippendorff’s alpha, which accepts a variable number of ratings per unit without forcing listwise deletion, is the better fit. See CASRAI’s guide to Krippendorff’s alpha for that calculation.
Computing it without doing the arithmetic by hand
- R — the
irrpackage’skappam.fleiss()function takes a subjects-by-raters matrix of category labels and returns kappa directly, with adetail = TRUEoption that also reports a category-by-category kappa breakdown. - Python —
statsmodels.stats.inter_rater.fleiss_kappa()takes the N×k count table shown in the worked example above (subjects by categories, cell values are rater counts) rather than a raw ratings matrix, so the table needs to be tallied into that shape first. - SPSS, SAS and Stata — none has a dedicated Fleiss’ kappa command built in; each requires either a user-contributed macro/module (Stata’s community-contributed
kappaetccovers Fleiss’ kappa alongside Cohen’s) or computing the count table and applying the formula above directly, e.g. in a syntax/data step. - ReCal — the same free browser-based calculator referenced on CASRAI’s Krippendorff’s alpha guide reports Fleiss’ kappa alongside Cohen’s kappa and Krippendorff’s alpha from a pasted rating matrix, without installing anything.
Whichever tool is used, confirm it is computing Fleiss’ kappa specifically and not silently falling back to a two-rater Cohen’s kappa on the first two columns — a mistake that is easy to make invisible in a spreadsheet formula and produces a plausible-looking but wrong number.
What to report
- The coefficient by name — “Fleiss’ kappa”, not “kappa” alone, since a reader cannot otherwise tell it apart from Cohen’s kappa.
- The number of raters per subject (n) and number of subjects (N), and whether the same raters rated every subject or the pool rotated.
- The number of categories and their overall prevalence (the pj values), given the prevalence paradox above.
- The point estimate, and a confidence interval where sample size allows one to be computed meaningfully — Fleiss (1971) gives an asymptotic standard error formula, and most software that reports kappa reports it alongside.
- The raw percent agreement as a plain-language companion figure, not a replacement for kappa.
- The threshold or convention applied to interpret the number, stated as the convention it is rather than a validated cutoff.
The reliability reporting checklist (GRRAS) referenced on CASRAI’s coefficient-selection guide applies to Fleiss’ kappa the same way it does to Cohen’s kappa, Krippendorff’s alpha and the ICC.
Frequently asked questions
How many raters do I need for Fleiss’ kappa?
At least three, since two raters is exactly the case Cohen’s kappa already covers (and where Fleiss’ kappa reduces to the related but distinct Scott’s pi, not to Cohen’s kappa). There is no fixed upper limit; the formula generalises to any fixed number of raters per subject.
Do all subjects need to be rated by the same raters?
No — that is the specific problem Fleiss’ kappa was built to handle. Every subject needs the same number of raters, but the individual raters can differ from subject to subject, which is the normal case for a coding project staffed from a larger pool.
What if the number of raters per subject varies?
Fleiss’ kappa assumes a fixed n. If the count of raters genuinely differs by subject, use Krippendorff’s alpha instead, which is built to accept a variable and even incomplete rating matrix without discarding subjects.
Is 0.449 a good Fleiss’ kappa?
By the Landis & Koch convention it falls in the “moderate” band, but that convention is explicitly arbitrary and was written for the two-rater case before being carried over by common practice, not derived for Fleiss’ kappa specifically. Whether 0.449 is adequate depends on the stakes of the coding decision, the category prevalence (check the prevalence paradox above), and any threshold specified in the study’s own protocol before data collection.
Can I compute Fleiss’ kappa for ordinal categories with weighting, like weighted Cohen’s kappa?
Not in Fleiss’ original 1971 formulation, which treats all categories as nominal and any disagreement as equally severe. For ordinal, multi-rater data where the size of a disagreement should matter, Krippendorff’s alpha with its ordinal difference function, or an ICC on the ratings treated as numeric, are the standard alternatives — see CASRAI’s guide to choosing an inter-rater reliability coefficient for the full decision logic.
This guide sits in CASRAI’s research methods cluster. For the full coefficient-selection logic this page assumes, see inter-rater reliability: choosing a coefficient; for the variable-rater alternative, see Krippendorff’s alpha; for continuous ratings, see the intraclass correlation coefficient; and for the underlying measurement-theory background, see reliability in research measurement.
