Written and maintained by CASRAI Editorial Board
Published
On this page
Document analysis is a systematic qualitative research method in which documentary materials—both physical records and digital texts—are critically appraised and synthesized to elicit meaning, gain understanding, and develop empirical knowledge. As articulated by Glenn A. Bowen (2009), it combines elements of content analysis and thematic analysis to identify patterns, corroborate findings, and establish contextual grounding across research inquiries.
Definition of document analysis
Document analysis is a formal research procedure for reviewing and evaluating printed and electronic documents, including public records, institutional reports, policy manuals, strategic plans, and meeting minutes. Unlike casual background reading, document analysis relies on purposeful sampling, systematic quality appraisal, and structured coding. It functions both as an independent methodology and as a central vehicle for triangulation in qualitative and mixed-method studies.
In social and health sciences research, documents provide stable, non-reactive data. By pairing document analysis with interviews, investigators verify participant accounts and uncover institutional trajectories. Rigorous approaches to analyzing qualitative data treat documentary texts with the same analytical precision applied to transcripts in a qualitative descriptive research design.
Methodological framework and intercoder agreement formula
Rigorous document analysis begins with quality appraisal. Sociologist John Scott (1990) established four foundational criteria for evaluating documentary evidence prior to textual coding:
- Authenticity: Confirms the document is genuine, complete, and of verifiable provenance and authorship.
- Credibility: Assesses whether contents are accurate and free from deliberate distortion or organizational spin.
- Representativeness: Evaluates whether available documents are typical of their class, accounting for archival survival bias.
- Meaning: Ensures the document's literal terminology and contextual nuances are clearly comprehensible.
Following appraisal, researchers apply Bowen's (2009) three-step progression: superficial examination (skimming), thorough examination (in-depth reading), and interpretive coding. Coding integrates quantitative content analysis (tracking category frequencies) with qualitative thematic analysis (identifying latent themes), often utilizing axial coding or structured template analysis.
To establish reliability when multiple investigators code a documentary corpus, methodologists evaluate intercoder agreement. Miles and Huberman (1994) formulated the foundational percentage of agreement metric:
$$text{Reliability} = frac{text{Agreements}}{text{Agreements} + text{Disagreements}}$$
Where:
- Agreements (A): The number of text passages where independent coders assign the identical category code.
- Disagreements (D): The number of passages where coders assign divergent codes.
Similarly, Holsti (1969) defined the observed proportion of agreement ($PA_o$):
$$PA_o = frac{2M}{n_1 + n_2}$$
Where $M$ is mutually agreed decisions, and $n_1$ and $n_2$ are total coding decisions by Coder 1 and Coder 2.
Because raw agreement does not adjust for chance overlap, researchers calculate Cohen's Kappa (κ):
$$kappa = frac{P_o – P_e}{1 – P_e}$$
Where $P_o$ is observed agreement ($A / N$), and $P_e$ is expected chance agreement: $P_e = sum_{i=1}^{k} (R_i / N times C_i / N)$, with $R_i$ and $C_i$ representing category marginal totals, and $N$ representing total text units.
Worked example: Coding an institutional policy corpus
An empirical study examined research governance across 10 universities. Two independent analysts coded $N = 120$ policy excerpts using four categories: Data Governance and Retention (DGR), Authorship and Credit (ACA), Conflict of Interest (COI), and Non-Compliance Procedures (NIP), tracked via computer-assisted qualitative data analysis software.
The cross-tabulation of independent decisions is shown below:
| Coder A | Coder B: DGR | Coder B: ACA | Coder B: COI | Coder B: NIP | Total Coder A |
|---|---|---|---|---|---|
| DGR | 38 | 0 | 4 | 0 | 42 |
| ACA | 3 | 28 | 0 | 0 | 31 |
| COI | 0 | 0 | 24 | 3 | 27 |
| NIP | 0 | 0 | 4 | 16 | 20 |
| Total Coder B | 41 | 28 | 32 | 19 | 120 |
Step 1: Calculate Observed Agreements and Disagreements
Agreements on diagonal: $A = 38 + 28 + 24 + 16 = 106$. Disagreements off-diagonal: $D = 4 + 3 + 3 + 4 = 14$. Total units: $N = 106 + 14 = 120$.
Step 2: Calculate Miles and Huberman Percent Agreement
$$text{Reliability} = frac{106}{106 + 14} = frac{106}{120} approx 0.8833 quad (88.33%)$$
Holsti's formula yields identical observed agreement: $PA_o = (2 times 106) / (120 + 120) = 212 / 240 approx 0.8833$.
Step 3: Calculate Expected Agreement ($P_e$)
Marginal probabilities per category: DGR $(42/120) times (41/120) = 0.1196$; ACA $(31/120) times (28/120) = 0.0603$; COI $(27/120) times (32/120) = 0.0600$; NIP $(20/120) times (19/120) = 0.0264$. Expected chance agreement: $P_e = 0.1196 + 0.0603 + 0.0600 + 0.0264 = 0.2663$.
Step 4: Compute Cohen's Kappa (κ)
$$kappa = frac{0.8833 – 0.2663}{1 – 0.2663} = frac{0.6170}{0.7337} approx 0.8409 quad (kappa = 0.841)$$
How to interpret document analysis findings
Interpreting document analysis requires evaluating both statistical coding reliability and qualitative evidence convergence.
| Metric or Criterion | Observed Value | Standard Threshold | Methodological Interpretation |
|---|---|---|---|
| Percent Agreement | 88.33% | ≥ 80.0% (Miles & Huberman) | High coding consistency; operational definitions are stable across coders. |
| Cohen's Kappa (κ) | 0.841 | 0.81 – 1.00 (Landis & Koch) | Almost perfect agreement beyond chance; robust codebook reliability. |
| Scott's Authenticity | Confirmed | Verified provenance | Records originate from verified institutional governance offices. |
| Scott's Credibility | Contextualized | Audience awareness | Policies capture formal organizational rules; requires practice triangulation. |
| Scott's Representativeness | Complete corpus | Documented sampling | Captures 100% of publicly available research governance policies in target sample. |
Qualitative interpretation examines how documentary evidence converges with or diverges from other data. If institutional policies require formal data sharing but researcher interviews reveal low compliance, this divergence highlights a critical policy-practice gap that neither method could uncover alone.
Assumptions, validity criteria, and common mistakes
Valid document analysis rests on core qualitative assumptions and trustworthiness standards (Lincoln & Guba, 1985):
- Social Construction: Documents are purposeful communications shaped by institutional priorities rather than unmediated objective truth.
- Triangulation: Documentary findings achieve credibility when corroborated across multiple document types and field data.
- Audit Trails: Dependability requires maintaining a detailed search log, document retrieval dates, and version records.
- Reflexivity: Researchers must document theoretical assumptions that guide category selection and text interpretation.
Analysts should avoid four common methodological pitfalls:
- Treating Documents as Infallible Truth: Accepting official texts at face value without evaluating authorial purpose and omitted perspectives.
- Overlooking Survival Bias: Analyzing only accessible texts without considering what unpreserved or restricted files might reveal.
- Imprecise Codebook Boundaries: Employing vague category definitions without clear inclusion and exclusion criteria.
- Decontextualized Quoting: Extracting passages without considering the broader document context and intended institutional readership.
How to report document analysis in APA format
Method sections should detail corpus boundaries, appraisal steps, coding procedures, and reliability statistics. Below is an APA 7th reporting model:
Method: Documentary Data Collection and Analysis
A systematic document analysis was conducted across 10 institutional research policies ($N = 120$ analyzed passages). Document appraisal followed Scott's (1990) criteria (authenticity, credibility, representativeness, and meaning). Following Bowen's (2009) dual-stage framework, documents underwent preliminary scanning followed by in-depth qualitative coding combining deductive policy categories with inductive thematic generation. Two independent analysts coded the 120-unit corpus. Intercoder percentage agreement was 88.33%, exceeding the 80% benchmark recommended by Miles and Huberman (1994). Chance-corrected reliability was $kappa = .841$, indicating almost perfect agreement (Landis & Koch, 1977). All 14 discrepancies were resolved through consensus review, with codebook adjustments documented in the project audit log.
How to calculate intercoder agreement in Excel, R, and Python
Researchers can verify intercoder metrics using standard computational tools.
1. Excel Formulas
With contingency values in B2:E5, row totals in F2:F5, column totals in B6:E6, and total units $N = 120$ in F6:
- Observed Agreement ($P_o$):
=(B2+C3+D4+E5)/F6(returns0.8833) - Expected Agreement ($P_e$):
=SUMPRODUCT(F2:F5, B6:E6)/(F6^2)(returns0.2663) - Cohen's Kappa (κ):
=(Cell_Po - Cell_Pe)/(1 - Cell_Pe)(returns0.8409)
2. Base R Script
m <- matrix(c(38,0,4,0, 3,28,0,0, 0,0,24,3, 0,0,4,16), nrow=4, byrow=TRUE)
n <- sum(m)
po <- sum(diag(m)) / n
pe <- sum((rowSums(m)/n) * (colSums(m)/n))
kappa <- (po - pe) / (1 - pe)
cat(sprintf("Observed Agreement: %.2f%%n", po * 100))
cat(sprintf("Cohen's Kappa: %.4fn", kappa))
3. Python Script
table = [[38,0,4,0], [3,28,0,0], [0,0,24,3], [0,0,4,16]]
n = sum(sum(r) for r in table)
po = sum(table[i][i] for i in range(len(table))) / n
pe = sum((sum(r)/n) * (sum(table[i][j] for i in range(len(table)))/n) for j, r in enumerate(table))
kappa = (po - pe) / (1 - pe)
print(f"Observed Agreement: {po:.4f} ({po * 100:.2f}%)")
print(f"Cohen's Kappa: {kappa:.4f}")
Frequently asked questions
What is the difference between document analysis and a literature review?
A literature review synthesizes published scholarship to identify theoretical gaps and establish a background rationale for a study. In contrast, document analysis treats documents as primary empirical data to be sampled, systematically coded, and analyzed to answer specific empirical research questions.
How does document analysis differ from qualitative content analysis?
Document analysis is an overarching research methodology encompassing document sourcing, provenance appraisal (evaluating authenticity and credibility), and contextual interpretation. Content analysis is an analytical coding technique used within document analysis to systematically categorize textual passages and track frequency patterns.
How many documents are needed for a qualitative document analysis?
Sample sizes are determined by data saturation rather than statistical power formulas. Researchers continue sampling until analyzing additional documents yields no new conceptual categories or insights. For bounded organizational case studies, 10 to 30 core policy manuals or annual reports typically provide sufficient depth.
Can document analysis serve as a standalone research method?
Yes. Document analysis functions effectively as an independent methodology when examining historical events, legislative development, or organizational governance where human participants are unavailable. However, contemporary social research frequently combines documents with interviews to facilitate triangulation.
Do researchers need institutional review board (IRB) approval for document analysis?
In most jurisdictions, analyzing publicly accessible government archives, policies, or published texts is exempt from IRB review because it does not involve living human subjects. Conversely, studies analyzing private correspondence, student files, or confidential medical records require formal ethics review and institutional clearance.
Sources
- Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27–40. https://doi.org/10.3316/QRJ0902027 (Read 2026-11-08).
- Holsti, O. R. (1969). Content Analysis for the Social Sciences and Humanities. Reading, MA: Addison-Wesley (Read 2026-11-08).
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310 (Read 2026-11-08).
- Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Beverly Hills, CA: SAGE Publications (Read 2026-11-08).
- Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis: An Expanded Sourcebook (2nd ed.). Thousand Oaks, CA: SAGE Publications (Read 2026-11-08).
- O'Leary, Z. (2014). The Essential Guide to Doing Your Research Project (2nd ed.). London: SAGE Publications (Read 2026-11-08).
- Scott, J. (1990). A Matter of Record: Documentary Sources in Social Research. Cambridge: Polity Press (Read 2026-11-08).
