Reliability is the consistency of a measurement, while validity is whether the measurement captures what it is intended to capture. Together they are the two pillars of psychometrics. A psychological test is only as trustworthy as these properties allow, and reporting them is a basic expectation of credible, reproducible research.
The three faces of reliability
Reliability concerns whether a measure gives consistent results. It comes in several forms depending on the source of consistency being examined:
- Test-retest reliability: do the same people get similar scores when measured again after a delay? High test-retest reliability suggests the instrument captures a stable attribute rather than transient noise.
- Inter-rater reliability: when human raters score the same behaviour, do they agree? Strong inter-rater reliability shows that the result reflects the thing observed, not the observer.
- Internal consistency: do items on a scale that are meant to measure one construct correlate with each other? This is commonly summarised by Cronbach’s alpha, which indexes how well a set of items hang together.
The three faces of validity
Validity concerns meaning—whether the score corresponds to the intended construct. The main types are:
- Construct validity: does the test actually measure the abstract concept it targets, such as anxiety or numerical ability? Evidence accumulates from how scores relate to other measures as theory predicts.
- Content validity: do the items adequately sample the full domain? A maths test that only covered addition would have poor content validity for general numeracy.
- Criterion validity: does the score predict or correspond to an external benchmark, such as later performance or an established gold-standard measure?
Reliability and validity at a glance
| Property | Type | Key question |
|---|---|---|
| Reliability | Test-retest | Are scores stable over time? |
| Reliability | Inter-rater | Do different raters agree? |
| Reliability | Internal consistency (Cronbach’s alpha) | Do items measure one thing together? |
| Validity | Construct | Does it measure the intended concept? |
| Validity | Content | Do items cover the whole domain? |
| Validity | Criterion | Does it predict a relevant outcome? |
Why a measure can be reliable but not valid
This is the most important conceptual point in psychometrics, and it is worth stating carefully. Reliability is necessary but not sufficient for validity. A bathroom scale that always reads three kilograms heavy is perfectly reliable—it gives the same answer every time—yet it is not a valid measure of weight, because it is consistently wrong. Likewise, a personality questionnaire can produce stable scores that nonetheless do not correspond to the trait it claims to assess. A measure cannot be valid without being reliable, but it can be reliable without being valid. Validity is therefore the higher bar. The practical implication is that demonstrating consistency is only the first step; an instrument must additionally be shown to track the construct it names before its scores can support any substantive claim.
How reliability is estimated in practice
Each form of reliability has a characteristic study design. Test-retest reliability is estimated by administering the same measure to the same people twice and correlating the two sets of scores; the delay must be long enough that memory of the first sitting does not inflate agreement, but short enough that the trait itself has not genuinely changed. Inter-rater reliability is assessed by having two or more trained raters score the same material independently and computing their agreement, often with a coefficient that corrects for chance. Internal consistency is calculated from a single administration by examining how the items intercorrelate, with Cronbach’s alpha the most familiar summary. Reporting which coefficient was used, and its value, lets readers judge whether a measure is fit for purpose.
A note on Cronbach’s alpha
Alpha is ubiquitous but frequently misread. A high value does not by itself prove a scale measures a single construct; it is sensitive to the number of items, so long scales can post a respectable alpha even when their items are only loosely related. Conversely, a very high alpha may signal redundant, near-duplicate items rather than a well-rounded measure. Alpha is therefore best treated as one piece of evidence about internal structure, interpreted alongside the scale’s design and its factor structure, not as a single pass-or-fail threshold.
Validity is an accumulating argument
Modern psychometrics treats validity less as a fixed property a test “has” and more as an evidence-based argument that builds over time. Construct, content and criterion evidence each contribute, and a measure earns confidence as independent studies show its scores behaving as theory predicts—correlating with related measures, diverging from unrelated ones and predicting relevant outcomes. This framing explains why a brand-new instrument cannot simply be declared valid; validity is demonstrated through replication, which ties measurement quality directly to the field’s reproducibility agenda.
Implications for research and assessment
These properties are not academic niceties; they determine whether a finding will replicate. Instruments with poor reliability add noise that can mask real effects or generate spurious ones, a concern at the heart of the field’s work on reproducibility. Many critiques of popular tools reduce to validity questions—for example, the measurement objections to the Myers-Briggs Type Indicator concern reliability and construct validity. Sound responsible assessment requires that both properties be measured and disclosed.
Reliability, error and the individual score
Reliability has a direct, practical meaning for how much trust to place in a single person’s score. Every observed score can be thought of as a true score plus measurement error, and the lower the reliability, the larger that error band. The standard error of measurement translates a reliability coefficient into a margin of uncertainty around an individual’s result, which is why responsible test reports present scores as ranges rather than precise points. Ignoring this band is a common misuse: treating a one-point difference between two people as meaningful when it falls well within measurement error. For consequential decisions, the size of the error band can matter as much as the score itself, and it should be reported alongside the headline number.
Reporting psychometrics transparently
Researchers should report which reliability and validity evidence supports each measure, ideally with the relevant coefficients. Consistent terminology helps: defining terms in a shared research dictionary lets readers compare studies, and clear guidance for authors turns good intentions into routine practice. Transparency about measurement is one of the cheapest ways to improve the reliability of the literature as a whole.
Frequently asked questions
What is the difference between reliability and validity?
Reliability is consistency—getting the same result repeatedly—while validity is accuracy—measuring the intended construct. A test must be reliable to be valid, but reliability alone does not guarantee validity.
Can a test be reliable but not valid?
Yes. A scale that consistently reads three kilograms too heavy is reliable but not valid. The result is stable yet systematically wrong, so it does not measure true weight.
What is Cronbach’s alpha?
Cronbach’s alpha is a common index of internal consistency. It estimates how well the items on a scale that are meant to measure one construct correlate with one another.
Why do reliability and validity matter for reproducibility?
Measures with weak reliability or validity add noise and bias, making findings harder to replicate. Reporting these properties is part of producing reproducible, trustworthy research.
Leave a Reply