Written and maintained by CASRAI Editorial Board
Last updated
Intra-rater reliability is the consistency of the measurements or ratings that one rater produces when scoring the same subjects on two or more occasions. Koo and Li (2016) put it as the variation of data measured by one rater across two or more trials. For a continuous measure, the standard statistic is the intraclass correlation coefficient (ICC), with trials taking the place of raters; for categorical ratings it is Cohen’s kappa (κ).
Definition: what intra-rater reliability measures
Intra-rater reliability answers one question: if the same person scores the same things again, do they get the same answer? McHugh (2012) describes it as the reliability of a single data collector. The raters in the design are the same person every time, so the variation you are estimating is that person’s own drift or noise, plus any real change in the subject if the retest is not close enough in time.
Intra-rater vs inter-rater vs test-retest reliability
- Intra-rater: one rater, repeated trials on the same subjects. The question is the rater’s self-consistency.
- Inter-rater: two or more raters scoring the same subjects. The question is agreement between people. See choosing an inter-rater coefficient and the test-retest vs inter-rater comparison.
- Test-retest: the same instrument or test given twice, usually to the same respondents. The retest interval and drift are covered in test-retest reliability.
In practice an intra-rater study is often a test-retest design in which the “test” is a human judgement, so the two share the same ICC machinery. For the wider picture of reliability types, see reliability in research measurement.
Which ICC to use for intra-rater reliability
Koo and Li (2016) give three choices for a one-rater, repeated-trials design:
- Model: a two-way mixed-effects model, which they state is appropriate for testing intra-rater reliability with multiple scores from the same rater.
- Definition: absolute agreement, which they say should always be chosen for both test-retest and intra-rater studies. Consistency ignores a systematic shift between trials; absolute agreement penalises it.
- Type: single measurement if one rating per subject will be used in practice, or mean of k if the average of k ratings will be used. Koo and Li say this depends on how the measurement protocol will be applied.
For the full six-form menu and how the forms are labelled in different software, see the intraclass correlation coefficient guide.
Formula with every symbol explained
The two-way absolute-agreement ICC for a single measurement, written ICC(A,1), is computed from a two-way ANOVA table with subjects as rows and trials as columns:
ICC(A,1) = (MSR − MSE) / (MSR + (k − 1) × MSE + k × (MSC − MSE) / n)
- n: the number of subjects (rows).
- k: the number of trials by the same rater (columns).
- MSR: mean square for subjects, the between-subject variation.
- MSC: mean square for trials, which captures any systematic shift between trial 1 and trial 2.
- MSE: mean square error, the residual variation left after removing subjects and trials.
The consistency version drops the trial term: ICC(C,1) = (MSR − MSE) / (MSR + (k − 1) × MSE). The mean-of-k version for absolute agreement is ICC(A,k) = (MSR − MSE) / (MSR + (MSC − MSE) / n).
Worked example with the arithmetic shown
One rater scores 10 subjects on a 0 to 100 scale in two sessions. Session 1 scores are 12, 15, 18, 21, 24, 27, 30, 33, 36, 39. Session 2 scores are 14, 16, 21, 22, 27, 29, 33, 35, 38, 42. So n = 10 and k = 2.
- Session means: 25.5 and 27.7. Grand mean: 26.6.
- Sum of squares for subjects (SSR) = 1,535.8, so MSR = 1,535.8 / 9 = 170.644.
- Sum of squares for trials (SSC) = 10 × ((25.5 − 26.6)² + (27.7 − 26.6)²) = 10 × 2.42 = 24.2, so MSC = 24.2 / 1 = 24.2.
- Total sum of squares = 1,562.8. Error sum of squares (SSE) = 1,562.8 − 1,535.8 − 24.2 = 2.8, so MSE = 2.8 / 9 = 0.311.
- ICC(A,1) = (170.644 − 0.311) / (170.644 + 1 × 0.311 + 2 × (24.2 − 0.311) / 10) = 170.333 / 175.733 = 0.969.
- For comparison, ICC(C,1) = 170.333 / (170.644 + 0.311) = 0.996, and ICC(A,2) = 170.333 / (170.644 + 23.889 / 10) = 0.984.
The gap between 0.969 and 0.996 comes entirely from the second session reading about 2.2 points higher on average. Absolute agreement counts that drift against the rater; consistency does not. Your software also reports a 95% confidence interval, which is what you should interpret (below).
The categorical version: Cohen’s kappa
If the rater assigns categories, use kappa. Say one rater classifies 40 images as positive or negative twice. Both times positive: 18. Positive then negative: 4. Negative then positive: 2. Both times negative: 16.
- Observed agreement = (18 + 16) / 40 = 0.85.
- First-pass totals: 22 positive, 18 negative. Second-pass totals: 20 positive, 20 negative.
- Chance agreement = (22 × 20 + 18 × 20) / 40² = 800 / 1,600 = 0.50.
- κ = (0.85 − 0.50) / (1 − 0.50) = 0.70.
Percent agreement alone (85%) overstates reliability because, as McHugh (2012) notes, it does not take account of the possibility that raters guessed on scores. Kappa corrects for that. See Cohen’s kappa for two raters for the formula in detail.
Interpretation: reading the result
Koo and Li (2016) recommend basing the label on the 95% confidence interval of the ICC estimate, not the estimate itself.
| Statistic | Value | Label | Source |
|---|---|---|---|
| ICC | Below 0.50 | Poor | Koo and Li (2016) |
| ICC | 0.50 to 0.75 | Moderate | Koo and Li (2016) |
| ICC | 0.75 to 0.90 | Good | Koo and Li (2016) |
| ICC | Above 0.90 | Excellent | Koo and Li (2016) |
| Kappa (Cohen) | 0.01 to 0.20 | None to slight | McHugh (2012) |
| Kappa (Cohen) | 0.21 to 0.40 | Fair | McHugh (2012) |
| Kappa (Cohen) | 0.41 to 0.60 | Moderate | McHugh (2012) |
| Kappa (Cohen) | 0.61 to 0.80 | Substantial | McHugh (2012) |
| Kappa (Cohen) | 0.81 to 1.00 | Almost perfect | McHugh (2012) |
McHugh (2012) also offers a stricter reading for kappa, in which 0.60 to 0.79 is “moderate” (35 to 63% of data reliable) and 0.80 to 0.90 is “strong” (64 to 81%). In the worked example, an ICC point estimate of 0.969 would be labelled excellent only if the lower end of its confidence interval also exceeds 0.90, and κ = 0.70 is substantial on Cohen’s scale but moderate on McHugh’s.
Assumptions and common mistakes
- Choosing consistency instead of absolute agreement. Koo and Li say absolute agreement should always be chosen for intra-rater studies. A drift between sessions then shows up in the result.
- Mismatching the type to the use. Pick single or mean-of-k according to how the measurement will be taken in real use, not by which number looks higher.
- Judging by the point estimate. Label the result from the 95% confidence interval.
- Reporting percent agreement only. It ignores chance agreement (McHugh, 2012).
- Treating intra-rater as inter-rater. A high intra-rater value says one person is self-consistent; it does not say that two people would agree.
How to report intra-rater reliability (APA style)
State the design, the ICC form, the software, and the interval. Example wording:
Intra-rater reliability was assessed with one rater scoring 10 participants on two occasions. An ICC was calculated from a two-way mixed-effects model, absolute agreement, single measurement: ICC(A,1) = 0.97, 95% CI [lower, upper], using the interval reported by the software.
Reference-list entries follow the APA 7th edition journal-article pattern (author, year, sentence-case title, italic journal and volume, DOI):
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
How to calculate it in Excel, R or Python
Excel
Put the session 1 scores in A2:A11, session 2 in B2:B11, per-subject means in C2:C11 (=AVERAGE(A2:B2)), and the two session means in A13:B13. Then:
MSR =2*DEVSQ(C2:C11)/9
MSC =10*DEVSQ(A13:B13)/1
MSE =(DEVSQ(A2:B11)-2*DEVSQ(C2:C11)-10*DEVSQ(A13:B13))/9
ICC(A,1) =(MSR-MSE)/(MSR+MSE+2*(MSC-MSE)/10)R
library(irr)
x <- cbind(s1 = c(12,15,18,21,24,27,30,33,36,39),
s2 = c(14,16,21,22,27,29,33,35,38,42))
icc(x, model = "twoway", type = "agreement", unit = "single")
# categorical: kappa2(cbind(first, second))Python
import numpy as np
x = np.array([[12,14],[15,16],[18,21],[21,22],[24,27],
[27,29],[30,33],[33,35],[36,38],[39,42]], dtype=float)
n, k = x.shape
gm = x.mean()
ssr = k*((x.mean(axis=1)-gm)**2).sum()
ssc = n*((x.mean(axis=0)-gm)**2).sum()
sse = ((x-gm)**2).sum() - ssr - ssc
msr, msc, mse = ssr/(n-1), ssc/(k-1), sse/((n-1)*(k-1))
icc_a1 = (msr-mse)/(msr+(k-1)*mse+k*(msc-mse)/n)
print(round(icc_a1, 3)) # 0.969All three use the same data as the worked example and return 0.969 for ICC(A,1).
FAQ
What is the difference between intra-rater and inter-rater reliability?
Intra-rater reliability compares one rater with themselves across repeated trials. Inter-rater reliability compares different raters with each other. The same coefficients can be used, but the question and the design differ.
Which ICC should I use for intra-rater reliability?
Koo and Li (2016) recommend a two-way mixed-effects model with absolute agreement, and single or mean-of-k type depending on how the measurement will be used.
What is a good intra-rater reliability value?
For ICC, Koo and Li label values below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good and above 0.90 excellent, applied to the 95% confidence interval.
Can I use percent agreement?
It is easy to compute but it does not account for raters guessing (McHugh, 2012), so report kappa for categorical ratings.
Sources
- Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://pmc.ncbi.nlm.nih.gov/articles/PMC4913118/ (checked 2026-10-29).
- McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/ (checked 2026-10-29).
