Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.
Research Methods & Statistics

Measurement, Reliability & Validity

A study is only as good as its measurements. This sub-cluster covers the psychometric concepts that determine whether an instrument actually measures what it claims to: reliability (consistency of measurement, including Cronbach's alpha for internal consistency and test-retest reliability for stability over time) and validity (whether an instrument measures the intended construct, including content, construct, and criterion validity). It also covers instrument development and adaptation — how existing scales are chosen, modified, or built from scratch, and what evidence is needed to justify that choice to reviewers and funders.

Guides

Test-Retest Reliability: Choosing the Retest Interval, and Checking for Drift

The retest interval is the design decision a test-retest coefficient cannot recover from. What the interval trades off, what COSMIN actually asks about it, which ICC form a test-retest design needs, and why only a mean difference or Bland-Altman check reveals systematic drift between occasions.

Content Validity: Does Your Instrument Cover the Whole Construct?

Content validity is whether an instrument’s items adequately sample the full construct domain, as judged by expert panels. Covers the expert-panel process, the content validity ratio (CVR), the content validity index (CVI), and the distinction from face validity, each worked through a hand-calculated illustrative example.

Levels of Measurement: Nominal, Ordinal, Interval and Ratio

A comparison of the four levels of measurement (nominal, ordinal, interval, ratio), the operations and statistics each one permits, and a step-by-step flowchart for classifying any variable.

Accuracy vs Precision in Measurement

Accuracy is closeness to the true value (systematic error); precision is closeness of repeated measurements to each other (random error). How to tell them apart, why “precise but inaccurate” is the dangerous case, and how each connects to reliability, validity, calibration, and diagnostic accuracy.

Convergent and Discriminant Validity

Convergent validity means a measure correlates with other measures of the same construct; discriminant validity means it does not correlate with measures of unrelated constructs. This guide covers both, the Campbell-Fiske multitrait-multimethod matrix that tests them together, and the modern AVE/Fornell-Larcker/HTMT thresholds.

Intraclass Correlation Coefficient (ICC): Forms, Interpretation, and How to Report It

What the ICC measures, the ICC(1,1)/(2,1)/(3,1) forms and when to use each, Koo & Li interpretation benchmarks, ICC vs. kappa/Pearson r/Bland-Altman/Cronbach’s alpha, sample size, and how to report results.

Visual Analogue Scale (VAS): Construction, Scoring and Minimal Important Change

A visual analogue scale is a 100 mm line with two anchors and no marks in between, scored by measured distance. This guide covers the construction rules that keep it valid, why VAS and NRS scores are not interchangeable, digital-versus-paper equivalence, and the published minimal important change values with the population, baseline severity and derivation method attached to each.

Criterion Validity: Concurrent and Predictive Evidence Explained

Criterion validity explained: concurrent vs. predictive evidence, how to choose a defensible criterion, criterion contamination, attenuation, and a worked correlation example.

Floor and Ceiling Effects: Detecting Range Compression and Fixing It

How floor and ceiling effects compress variance at a scale’s extremes, distort correlations, group comparisons and responsiveness, and the instrument-selection and statistical fixes for each.

Split-Half Reliability and the Spearman-Brown Correction

A split-half correlation is the reliability of a half-length test, not the full one, and different equally valid splits of the same items produce different numbers. What the Spearman-Brown correction fixes, what it can’t fix, and why Cronbach’s alpha — the average of every possible split — became the standard instead.

Cronbach’s Alpha: What It Measures, How to Interpret It, and When to Use Omega Instead

Cronbach’s alpha measures internal consistency, not unidimensionality or reliability in general — the most common misreading of the statistic. This guide covers interpretation, why the 0.7 threshold is misapplied, how adding items inflates alpha, when McDonald’s omega is the better choice, and how to report alpha correctly in a methods section.

The Observer Effect in Research: Reactivity, Not Physics

The observer effect (reactivity) is behavior change caused by awareness of being studied. Learn how it differs from the Hawthorne effect, demand characteristics, social desirability bias, and observer bias, where it bites hardest, and how to mitigate it responsibly.

Ecological Validity: Defending It Against the Trade-Off with Internal Validity

Ecological validity means a study’s setting and tasks resemble real-world conditions, and that its findings still hold outside the study. This guide covers what it actually asks, where it sits in the standard validity framework, the concrete design choices (setting, task realism, measurement obtrusiveness, assignment) that trade it off against internal validity, and how to defend it in a methods section.

Construct Validity: Definition, Evidence Types, and Threats

Construct validity is whether an instrument truly measures the theoretical construct it claims to. This guide covers the Messick/AERA-APA-NCME evidence-argument framing, convergent, discriminant, nomological, known-groups and factorial evidence, the multitrait-multimethod matrix, AVE/HTMT, and the two core threats: construct underrepresentation and construct-irrelevant variance.

Meaning in Life Questionnaire (MLQ): Scoring, Reliability and Permission

How to score the Meaning in Life Questionnaire (MLQ): reverse-code item 9, sum Presence and Search separately, with reliability and permission terms.

Score Test: Formula, Worked Example and Calculation

Understand the Rao score test through a worked proportion example, null-model formulas, interpretation, and Excel, R and Python calculations.

Flourishing Scale (Diener): Items, Scoring and Licence

The 8-item Flourishing Scale: items, 8-56 scoring with worked example, APA reporting, code, and licence terms compared with WEMWBS.

Barratt Impulsiveness Scale: Scoring and Norms

Score the Barratt Impulsiveness Scale (BIS-11), calculate 3 subscales, reverse-code items, and interpret scores with normative benchmarks and code examples.

Functional Activities Questionnaire (FAQ) Scoring

A psychometric guide to the Functional Activities Questionnaire: 10 IADL items, scoring formulas, clinical cutoffs (≥9), NACC norms, and analysis code.

Multidimensional Scale of Perceived Social Support

Guide to the Multidimensional Scale of Perceived Social Support (MSPSS): scoring formulas, subscales, cutoffs, worked examples, and APA reporting templates.

Normalize Z Score: Formulas, Calculation & Guide

A complete guide to z-score normalization: formulas, step-by-step worked arithmetic, reading guidance, and verified Excel, R, and Python code snippets.

Self-Compassion Scale: Items, Scoring & Norms

A psychometric guide to the Self-Compassion Scale (SCS): explore item subscales, reverse-scoring formulas, worked calculations, and APA 7th reporting norms.

Heart Rate Variability (HRV) Metrics: Time-Domain (SDNN, RMSSD), Frequency-Domain (LF, HF) & ESC/NASPE Standards

Complete clinical research reference to Heart Rate Variability (HRV) metrics: ESC/NASPE (1996) standards, time-domain (SDNN, RMSSD), frequency-domain, and ECG/PPG.

Calibration Curve in Machine Learning: Expected Calibration Error (ECE), Platt Scaling vs. Isotonic Regression, and Temperature Scaling

How to evaluate and recalibrate machine learning probability estimates: reliability diagrams, Expected Calibration Error (ECE), Platt scaling vs. isotonic regression, and temperature scaling.

Framingham Risk Score: 10-Year CVD Equations, Cox Regression & EHR Calculation

Biostatistical guide to the Framingham Risk Score (Wilson 1998 and D’Agostino 2008), Cox proportional hazards equations, ATP III thresholds, and EHR CDS.

Internal vs. External Validation of Clinical Prediction Models: Optimism Correction, TRIPOD+AI Hierarchy, and Analytical vs. Clinical Validity

A methodological comparison of internal validation (bootstrapping, cross-validation) and external validation (temporal, geographic, and domain validation) for clinical prediction models under the Steyerberg framework and TRIPOD+AI guidelines, alongside analytical validity, clinical validity, and clinical utility.

Harrell’s C-Index (Concordance Index): Right-Censored Survival Formula, Comparable Pairs, and Uno’s IPCW C-Statistic

A biostatistical guide to Harrell’s C-index (concordance index) for right-censored survival analysis, detailing comparable pair rules, tied risk score adjustments, censoring distribution bias, Uno’s IPCW C-statistic, and worked numerical calculations.

AUROC vs. AUPRC: ROC Curve vs. Precision-Recall Curve in Clinical Prediction and Rare-Event Screening

Biostatistical comparison of Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC), explaining class imbalance, prevalence-dependent baselines, and clinical alert evaluation.

C-Statistic (Concordance Statistic): Formula, AUROC Equivalence, and Harrell’s C-Index

The C-statistic (concordance statistic) measures the rank-order discrimination of a diagnostic or prognostic prediction model across binary logistic regression (AUROC) and right-censored survival models (Harrell’s C-index).

Calibration Slope, Calibration-in-the-Large (CITL), and ICI in Clinical Prediction Models

The calibration slope evaluates the spread of predicted risks relative to observed outcomes on the log-odds scale (target = 1.0), working alongside calibration-in-the-large (target = 0.0) and the Integrated Calibration Index (ICI) to assess clinical prediction model reliability across the Van Calster hierarchy.

Brief Resilience Scale (BRS): Scoring, Norms and Research Guide

Calculate and interpret Brief Resilience Scale (BRS) scores with reverse-coded items, normative benchmarks, APA reporting, and Excel, R, and Python code.

Strengths and Difficulties Questionnaire (SDQ): Scoring, Cut-Points and Research Use

How the 25-item SDQ is scored: item-to-scale map, reverse items, pro-rata rule, 3- and 4-band cut-points, a worked example and R/Python code.

PHQ-15: Scoring, Cutoffs and Psychometrics of the Somatic Symptom Scale

How the PHQ-15 somatic symptom scale is scored, what the 5, 10 and 15 cutpoints mean, the validity evidence, and how to calculate and report it.

SCARED Questionnaire: Scoring, Subscales and Cutoffs

How the 41-item SCARED is scored: five subscales, the total cutoff of 25, a worked example, APA reporting and Excel, R and Python code.

Bayley Scales of Infant Development: Bayley-4 Scoring, Norms and Interpretation

What the Bayley-4 measures, how it is scored, and how to turn composites into z-scores and percentiles, with edition comparisons and APA reporting.

Psychophysics: Definition, Thresholds and Methods

Psychophysics measures how physical stimuli map to sensation. Learn thresholds, JND, Fechner’s founding role and a table for choosing a method.

SCOFF Questionnaire: Items, Scoring and Validity

SCOFF questionnaire: the five items, how to score it, and how its accuracy compares between the 1999 study and a 2020 meta-analysis.

Wender Utah Rating Scale (WURS-25): Scoring, Cutoffs and Psychometrics

WURS-25 scoring explained: 25 items rated 0-4, totals of 0-100, published cutoffs and psychometric properties, with a worked example.

International Personality Item Pool (IPIP): Scales, Scoring and Citation

The IPIP is a free, public-domain bank of personality items. See the scales, how to score them, how reliability changes with length and how to cite one.

Propagating Errors: Formulas, Rules Table and Worked Examples

How to propagate errors through a calculation: a GUM-based rules table for sums, products and functions, two worked lab examples, Excel, R and Python code.

Analytical Sensitivity: Guide to LoD and CLSI

Master analytical sensitivity in lab testing: CLSI EP17 detection metrics, LoD formulas, verification protocols, and clinical diagnostic comparisons.

Disability-Adjusted Life Year (DALY): Calculation, Uses, and Comparison to QALY

Learn the mathematical construction of Disability-Adjusted Life Years (DALYs), how YLL and YLD are combined, and key distinctions from QALYs in research.

Iowa Gambling Task: Protocol, Scoring & Validity

The Iowa Gambling Task (IGT) evaluates decision-making under uncertainty, risk sensitivity, and prefrontal function using standardized card payoff schedules.

Gauge R&R: ANOVA Method, Formulas & Lab Guide

A rigorous guide to Gauge R&R: two-way ANOVA decomposition, variance formulas, AIAG acceptance criteria, worked arithmetic, and Excel, R, and Python code.

Healthy Eating Index (HEI): Components, Scoring and Methods

The Healthy Eating Index explained: HEI-2020’s 13 components with point standards, a worked scoring example and NCI’s four scoring methods.

Intra-Rater Reliability: Definition, ICC Formula and Worked Example

Intra-rater reliability is one rater’s consistency across repeated trials. Learn which ICC to use, a worked example, kappa, and how to report it.

Rank Order Scale Examples: Questions and Scoring

Rank order scale examples: question wording, survey platform formats, two scoring methods compared on one data set, and a Friedman test with code.

Maturation Threat to Internal Validity: Definition, Example, Control

Maturation threatens internal validity when natural change over time is mistaken for a treatment effect. Definition, worked example and how to rule it out.

How to Find the P-Value From a Z-Score

Convert a z-score to a p-value for left-, right- and two-tailed tests, with a worked example, APA reporting and Excel, R and Python code.

F1 Score Formula: How to Calculate It and What Is Good

F1 score formula with every symbol explained, a worked example, macro/micro averaging rules and an honest answer on what counts as a good F1.

Ratio Scale Examples: What It Is, Formula and Worked Example

Ratio scale examples, the a/b formula and a worked calculation, with a true-zero test and Excel, R and Python code.

Brier Score: Formula, Worked Example and How to Interpret It

The Brier score is the mean squared error of probability forecasts. Formula, worked example, the benchmark to beat, and Excel, R and Python code.

Deming Regression: The Parametric Alternative to Passing-Bablok

Deming regression is the parametric errors-in-variables alternative to Passing-Bablok: it also corrects for error in both measurement methods, but requires normally distributed errors and an estimate of the error-variance ratio between the two methods.

Passing-Bablok Regression: Why OLS Is the Wrong Tool for Method Comparison

Ordinary least squares assumes the x-variable is error-free, which is false when comparing two fallible measurement methods and biases the slope. Passing-Bablok regression is the non-parametric alternative built for this exact problem, and it complements rather than replaces Bland-Altman analysis.

Cohen’s Kappa for Two Raters: Formula and Worked Example

Cohen’s kappa is the chance-corrected agreement statistic for exactly two raters classifying nominal categories. Formula, a worked 2×2 example, Landis & Koch interpretation bands, and the kappa paradox explained.

Recall Bias: Causes, Direction, and How to Reduce It

Recall bias is differential memory error between cases and controls in a retrospective study. This guide covers why it happens, which way it can push a result, and the design fixes that actually reduce it.

Demand Characteristics: How Participants Guess Your Hypothesis and Change Their Behavior

Demand characteristics are cues that let participants guess a study’s hypothesis and adjust their behavior to match it. This guide covers Orne’s origin of the term, Weber and Cook’s four subject roles, how to detect it with a post-experimental inquiry, and how to design against it.

Test Equating and Linking

Equipercentile equating, IRT linking via Stocking-Lord and Haebara, and the anchor-item design rules that make scores from different test forms comparable.

Anchoring Vignettes: Correcting for Differential Item Functioning in Self-Reports

How anchoring vignettes correct differential item functioning in self-reported survey scales, using King and Hopkins’s response-consistency method.

Test Information Functions and Measurement Precision

Item information functions sum into a test information function that shows measurement precision at every trait level, not just one sample-average reliability number. This guide covers the IRT formulas, the SE(theta) = 1/sqrt(I(theta)) relationship, how to read a TIF curve, and how to use one to target a short form or evaluate an instrument at the range that matters.

Factor Scores: Methods and When Not to Use Them

Regression, Bartlett, and Anderson-Rubin factor-score methods compared with a reproducible worked example of factor-score indeterminacy, plus the specific cases where a sum or mean score is the better choice.

Item-Total Correlation in Scale Purification

The corrected item-total correlation diagnostic for scale purification: what it measures, the conventional 0.30 cutoff, and how it relates to alpha if item deleted, with a reproducible worked example.

Differential Item Functioning (DIF): Detecting Item-Level Bias with the Mantel-Haenszel Method

Differential item functioning (DIF) is an item-level fairness problem distinct from overall test bias or group score differences: it occurs when people at the same trait level respond to one item differently depending on group membership. This guide covers uniform vs. non-uniform DIF and walks through the Mantel-Haenszel detection procedure with a reproducible simulated example.

The Rasch Model: Item-Invariant Measurement and the Person-Item Map

The Rasch model is a one-parameter IRT model built around invariant measurement: item difficulty and person ability that hold constant regardless of who or what was used to estimate them. This guide covers specific objectivity, infit/outfit fit statistics, and how to read a person-item (Wright) map.

Choosing and Validating a Patient-Reported Outcome Measure (PROM)

A five-step checklist for choosing an existing patient-reported outcome measure rather than building one: matching the construct, finding candidates, checking population-specific validation evidence, confirming an MCID exists, and clearing licensing and translation.

Fleiss’ Kappa for Multiple Raters: Formula and Worked Example

The Fleiss’ kappa formula for three or more raters, a fully worked five-subject rating-matrix example, why it isn’t simply Cohen’s kappa for 3+ raters, and the Landis & Koch bands carried over from Cohen’s kappa.

Krippendorff’s Alpha: Calculating Intercoder Reliability

The observed-vs-expected disagreement formula, a worked example with missing data across three coders, Krippendorff’s own 0.667/0.800 benchmarks, a bootstrap confidence-interval method, and software (R, Python, SPSS/SAS/Stata, ReCal).

Item Response Theory for Research Scales

What item response theory is, how the item-characteristic curve and its difficulty and discrimination parameters work, what the 1PL, 2PL, and 3PL models add, and why IRT (unlike classical test theory) enables adaptive testing.

Bland-Altman Plots for Method Agreement: Why Correlation Isn’t the Right Tool

Why Pearson correlation cannot assess agreement between two measurement methods, how to read the bias line and limits of agreement on a Bland-Altman plot, how to construct one from a worked dataset, and how to detect proportional bias.

Minimal Clinically Important Difference (MCID): Anchor-Based vs. Distribution-Based Estimation

Compares anchor-based MCID estimation (global rating of change, mean-change and threshold/ROC approaches) against distribution-based estimation (the 0.5 SD rule, SEM-based rules), explains why the two families routinely disagree, and covers which a reviewer will trust more plus how MCID feeds sample-size calculation and responder analysis.

Inter-Rater Reliability: Choosing the Right Coefficient

A selection table from data type and rater design to the right agreement statistic – Cohen’s and weighted kappa, Fleiss’ kappa, ICC, Krippendorff’s alpha and Gwet’s AC1 – with the kappa paradox worked through three 2×2 tables that share identical 95% agreement.

Average Variance Extracted (AVE): Calculation, the 0.50 Threshold, and Discriminant Validity

AVE is the mean proportion of indicator variance a construct explains rather than error. This guide gives the calculation, the origin and conventional status of the 0.50 threshold, and the Fornell-Larcker and HTMT discriminant-validity tests that consume it, including why the methodological literature has moved away from Fornell-Larcker.

Minimal Detectable Change (MDC): SEM-Based Calculation and the MDC-vs-MCID Decision Rule

What MDC is, the three routes to a standard error of measurement and the assumptions each carries, the MDC95 = 2.77 x SEM calculation worked end to end, the individual-versus-group distinction, and the four-cell decision rule for judging a change against both MDC and MCID.

Cronbach’s Alpha: Reliability & Interpretation Guide

What Cronbach’s alpha does and does not tell you about a scale: the formula and its assumptions, why a high alpha is not evidence of unidimensionality, what the benchmark thresholds are worth, and how to report reliability in APA 7th-edition format.

Response Bias: The Main Types and How to Design Against Them

Response bias is systematic, direction-consistent error in self-report data. This guide covers the five main types — acquiescence, extreme responding, social desirability, recall bias, and order effects — with the specific design fix for each.

The Hawthorne Effect: Why Being Watched Changes the Data

The Hawthorne effect is the tendency for people to change behavior because they know they are being studied. Learn the Western Electric study history, what the Levitt & List and Jones re-analyses actually found, and how to detect and design against reactivity.

Social Desirability Bias: Why Respondents Tell You What You Want to Hear

What social desirability bias is, the self-deception vs. impression-management mechanisms behind it, where it hits hardest, and the countermeasures (indirect questioning, list experiments, randomized response, self-administration) that reduce it.

Psychometrics: How Researchers Measure Things You Can’t Observe

How psychometrics turns unobservable constructs into reliable, valid scores: the construct-to-score pipeline, classical test theory vs. IRT, and how instruments are built and evaluated.

Reliability in Research: What It Means and How to Assess It

What reliability means in research measurement, how it differs from validity, classical test theory, the four types of reliability, what degrades it, attenuation, and how much reliability is enough.

Types of Validity in Research: Measurement Validity vs Design Validity

Measurement validity (face, content, construct, criterion) and design validity (internal, external, statistical-conclusion) are two distinct families constantly conflated under one word. This guide separates them and covers the validity-reliability relationship.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →