Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
The tidyr package is a core R library developed within the tidyverse to reshape, clean, and structure rectangular research datasets into a standardized tidy format. In a tidy dataset, every column represents an individual variable, every row represents an independent observation, and every cell stores a single empirical value. Researchers use tidyr primarily to pivot tables between wide and long layouts, separate compound measurement strings into distinct variables, expand implicit missing observational combinations, and rectangle complex hierarchical lists or JSON responses into analysis-ready data frames.
What it is used for in research
Modern scientific data collection rarely outputs rectangular tables in an immediate state for statistical modeling. Laboratory instruments, electronic health record registries, ecological field surveys, and public census databases routinely format tabular data to optimize data entry or visual human scanning rather than automated computational processing. Common structural layouts include placing time points across column headers, recording multiple clinical assay readings inside a single text cell, or leaving unobserved experimental conditions out of tabular records entirely. In empirical research workflows, the tidyr package provides a mathematically consistent set of verbs to transform these heterogeneous layouts into tidy data structures.
The formal foundation of tidy data was established by Hadley Wickham in 2014, mapping Edgar F. Codd’s third normal form from relational database theory to statistical data analysis. Wickham identified five persistent structural violations that hinder statistical computing:
- Column headers contain data values rather than variable names (such as recording calendar years or clinical visit numbers as separate columns).
- Multiple variables are concatenated into a single column (such as storing patient sex, cohort identifier, and subject number in a single compound ID string).
- Variables are stored across both rows and columns (such as placing observation dates along rows while stacking temperature, blood pressure, and heart rate along an attribute column).
- Multiple types of observational units are combined into the same rectangular table (such as interweaving patient-level demographic traits with daily repeated vital sign measurements).
- A single observational unit is fragmented across multiple distinct tables or spreadsheets.
Within academic research, tidyr serves as the bridge between raw data acquisition and formal downstream analysis. Statistical modeling functions in base R (such as lm(), glm(), and lme4::lmer()) as well as visualization libraries like ggplot2 require data in long, tidy formats where each predictor and response constitutes a single named vector. By integrating tightly with the dplyr package in R and adhering to foundational tidy data rules for structuring research data, tidyr allows research teams to build reproducible, programmatic data cleaning pipelines that replace error-prone manual spreadsheet restructuring.
Beyond traditional pivoting between wide and long representations, modern research workflows rely on tidyr for two advanced architectural patterns: handling observational completeness and data rectangling. In clinical trials and longitudinal ecological censuses, non-events or unobserved combinations often result in implicit missing values—combinations of experimental factor levels that do not appear in the raw dataset. The complete() function makes these implicit gaps explicit by generating the full Cartesian product of experimental treatments and populating missing cells with standard NA indicators. Simultaneously, modern high-throughput biology and digital health platforms frequently ingest nested JSON or XML payloads from web Application Programming Interfaces (APIs). Through its rectangling functions, tidyr decomposes nested hierarchical lists into flat rectangular tibbles without loss of relational provenance.
How to install or access it
The tidyr package is released and distributed through the Comprehensive R Archive Network (CRAN), ensuring platform-specific binary stability and automated dependency resolution across Windows, macOS, and Linux operating systems. Because tidyr is a core component of the tidyverse ecosystem, researchers can install it either as a standalone package or alongside the full tidyverse suite.
To install the official, peer-reviewed stable release from CRAN, run the standard installation command inside an active R console or RStudio session:
# Install standalone tidyr from CRAN
install.packages("tidyr")
# Alternatively, install the complete tidyverse suite
install.packages("tidyverse")
CRAN release 1.3.2 requires R version 4.1.0 or higher. Installation pulls necessary core dependencies, including vctrs (for robust underlying vector mechanics), rlang (for tidy evaluation and non-standard evaluation), purrr, tibble, stringr, and tidyselect. For institutional research clusters operating behind enterprise firewalls or running Posit Package Manager (PPM) mirrors, administrators can pin specific repository snapshot dates to ensure exact reproducibility across collaborative scientific teams.
To access the development version containing recent experimental patches or prerelease functionality, researchers can compile the package directly from Posit Software’s official GitHub repository using pak or remotes:
# Install the development version from Posit's official GitHub
# Requires Rtools on Windows or Xcode Command Line Tools on macOS
install.packages("pak")
pak::pak("tidyverse/tidyr")
Once installed, load tidyr into your R session using the library() command:
library(tidyr)
library(dplyr) # Recommended companion library for pipeline data manipulation
Step-by-step task
To demonstrate a realistic research pipeline, this step-by-step walkthrough transforms a messy, multi-center longitudinal clinical study dataset into a fully tidy structure ready for mixed-effects linear regression. The raw dataset contains four typical structural defects: repeated clinical laboratory visits recorded across separate wide columns, subject identifiers merged with treatment group labels, missing records that conceal unmeasured patient visits, and inconsistent text delimiters.
Researchers can run this complete sequence directly in an R console to replicate every transformation.
Step 1: Construct the raw messy clinical research dataset
First, create an illustrative research data frame reflecting raw multi-visit clinical trial data as exported from a legacy electronic data capture spreadsheet:
# Step 1: Create messy clinical trial dataset
raw_clinical = tibble::tribble(
~patient_tag, ~age, ~cohort, ~lab_v1, ~lab_v2, ~lab_v3,
"CTRL_001_M", 45, "Control", 112, 118, 124,
"CTRL_002_F", 52, "Control", 130, 128, NA,
"DRUG_001_F", 49, "Active", 145, 132, 120,
"DRUG_002_M", 61, "Active", 160, NA, 138
)
print(raw_clinical)
In this initial format, the biomarker values measured at Visit 1, Visit 2, and Visit 3 are spread across columns lab_v1, lab_v2, and lab_v3. This violates the first rule of tidy data: visit timing is a variable, not three separate attributes. Additionally, patient_tag conflates three distinct variables: treatment cohort code, patient numeric identifier, and biological sex.
Step 2: Reshape from wide to long with pivot_longer()
Transform the multi-visit laboratory columns into two tidy columns: one representing the visit indicator and one representing the measured biomarker concentration:
# Step 2: Pivot wide visit columns into tidy long rows
long_clinical = raw_clinical %>%
pivot_longer(
cols = starts_with("lab_v"),
names_to = "visit",
names_prefix = "lab_v",
values_to = "biomarker_mg_dl",
values_drop_na = FALSE
)
print(long_clinical)
The cols argument uses tidyselect syntax to capture all columns beginning with the prefix lab_v. The names_prefix argument automatically strips this string prefix, leaving only clean visit index numbers in the newly created visit column.
Step 3: Disentangle compound identifiers with separate_wider_delim()
In modern tidyr releases (version 1.3.0 and newer), the functions separate_wider_delim(), separate_wider_position(), and separate_wider_regex() supersede the older separate() function. They provide strict error handling when delimiter frequencies vary across rows. Separate the patient_tag column into distinct, analysis-ready attributes:
# Step 3: Split compound patient identifier into distinct variables
separated_clinical = long_clinical %>%
separate_wider_delim(
cols = patient_tag,
delim = "_",
names = c("arm_code", "subject_id", "sex")
)
print(separated_clinical)
Each component of the compound string now populates its own typed column: arm_code, subject_id, and sex, satisfying Codd’s rule that each cell must store a single atomic value. When combined with the stringr text cleaning functions and the janitor package for data cleaning, text discrepancies are resolved rapidly across large cohorts.
Step 4: Expose and impute implicit missing observations with complete() and fill()
In longitudinal research, a patient dropping out or missing a scheduled clinical visit results in missing observational rows. In our sample data, Patient CTRL_002 has an explicit NA for visit 3, but if an entire record had been omitted from collection, it would constitute an implicit missing value. Ensure observational completeness across all patients and scheduled visits using complete():
# Step 4: Expand dataset to ensure all subjects have records for visits 1, 2, and 3
complete_clinical = separated_clinical %>%
complete(
nesting(subject_id, arm_code, sex, age, cohort),
visit = c("1", "2", "3")
) %>%
arrange(subject_id, visit)
print(complete_clinical)
If longitudinal records contain baseline covariates recorded only during initial triage (such as baseline body mass index or smoking status), the fill() verb carries those values forward across subsequent visits using the .direction = "down" parameter.
Step 5: Pivot to wide format for publication summary tables with pivot_wider()
While longitudinal modeling requires long data, journal reporting guidelines often require cross-tabulated summary tables presenting baseline versus final follow-up values side by side. Use pivot_wider() to produce structured wide tables:
# Step 5: Pivot selected visits to wide format for comparative summary display
summary_wide = complete_clinical %>%
select(subject_id, cohort, visit, biomarker_mg_dl) %>%
pivot_wider(
names_from = visit,
names_prefix = "visit_",
values_from = biomarker_mg_dl
)
print(summary_wide)
The resulting table presents each patient on a single row, with dedicated columns for visit_1, visit_2, and visit_3, ideal for reporting in publication manuscripts or calculating paired difference scores.
Output and how to read it
Understanding and verifying console output in tidyr requires familiarity with tibble printing conventions and metadata structures. When executing tidyr pipelines, R outputs an augmented data frame known as a tbl_df (tibble). Below is the exact structure produced by R when inspecting the result of Step 4 (complete_clinical):
# A tibble: 12 x 7
subject_id arm_code sex age cohort visit biomarker_mg_dl
[chr] [chr] [chr] [dbl] [chr] [chr] [dbl]
1 001 CTRL M 45 Control 1 112
2 001 CTRL M 45 Control 2 118
3 001 CTRL M 45 Control 3 124
4 001 DRUG F 49 Active 1 145
5 001 DRUG F 49 Active 2 132
6 001 DRUG F 49 Active 3 120
7 002 CTRL F 52 Control 1 130
8 002 CTRL F 52 Control 2 128
9 002 CTRL F 52 Control 3 NA
10 002 DRUG M 61 Active 1 160
11 002 DRUG M 61 Active 2 NA
12 002 DRUG M 61 Active 3 138
When reviewing tidyr output, researchers should systematically verify four key architectural indicators:
- Tibble Dimensions: The header line
# A tibble: 12 x 7confirms that the dataset contains exactly 12 rows and 7 columns. In our study design with 4 distinct patients evaluated across 3 visits, an observational count of 12 rows confirms thatcomplete()successfully verified structural completeness without omitting or duplicating records. - Column Type Tags: Directly beneath each column name, R prints the vector storage class in abbreviated brackets:
[chr]for character vectors,[dbl]for double-precision floating-point numeric measurements, and[int]for integers. Ensuring thatbiomarker_mg_dlis classified as[dbl]rather than[chr]confirms that non-numeric characters (such as stray measurement units) did not coerce numeric measurements into text during pivoting. - Explicit Missingness Representation: Rows 9 and 11 display explicit
NAindicators for missing visit recordings. Because the cell containsNArather than an empty string""or arbitrary placeholder code (such as -999), downstream modeling functions likeglm()can correctly apply default missing-data handling routines. - Invariant Row Uniqueness: In a verified tidy dataset, the combination of observational identifier variables (here,
subject_idandvisit) forms a unique primary key. No two rows share identical combinations of subject identifier and visit index.
To inspect column data types and underlying attributes programmatically, execute dplyr::glimpse(complete_clinical) or base R’s str(complete_clinical). Before running advanced models, researchers who are importing datasets in RStudio should always run diagnostic glimpse checks immediately after tidying raw files.
Troubleshooting
Even experienced researchers encounter predictable challenges when restructuring real-world scientific data. Below are four common failure modes encountered in tidyr workflows along with concrete, verified fixes.
1. Duplicate Key Warning and List-Columns in pivot_wider()
When running pivot_wider(), users frequently encounter the following warning message:
Warning message:
Values from `value` are not uniquely identified; output will contain list-cols.
* Use `values_fn = list` to suppress this warning.
* Use `values_fn = length` to count values.
Cause: The combination of columns retained as row identifiers (the implicit or explicit id_cols) does not uniquely identify each observation. When multiple observations share the same ID and wide column name, pivot_wider() cannot place two scalar values into a single cell, so it packs them into a nested list.
Fix: Identify and eliminate duplicate observational keys before widening. You can identify duplicate rows using janitor::get_dupes(data, id_col, names_from_col). Alternatively, if multiple readings reflect repeated technical replicates that should be averaged, supply an aggregation function to the values_fn parameter:
# Aggregate technical replicates by taking their arithmetic mean
clean_wide = raw_data %>%
pivot_wider(
id_cols = subject_id,
names_from = assay_name,
values_from = concentration,
values_fn = mean
)
2. Incompatible Column Types in pivot_longer()
Attempting to pivot multiple columns that have differing storage types generates a type mismatch error:
Error in `pivot_longer()`:
! Can't combine `score_baseline` [double] and `score_followup` [character].
Cause: In a wide table, one column may contain purely numeric values (such as 14.2), while another contains text annotations (such as below limit of detection or missing flags). Because a single tidy column in R must be homogenous in storage type, pivot_longer() refuses to combine them silently.
Fix: Harmonize vector types before pivoting. Use values_transform = as.character within pivot_longer() to coerce all target columns to character vectors, clean the text flags, and subsequently convert confirmed numeric observations back to numeric types:
# Harmonize variable types during pivoting using values_transform
harmonized_long = raw_data %>%
pivot_longer(
cols = starts_with("score_"),
names_to = "timepoint",
values_to = "score",
values_transform = as.character
)
3. Mismatched Delimiter Counts in separate_wider_delim()
When splitting text columns with separate_wider_delim(), records with missing or surplus delimiter characters cause execution halts:
Error in `separate_wider_delim()`:
! Expected 3 pieces in each element of `patient_tag`.
! 2 values had too few pieces.
Fix: Modern tidyr verbs provide the too_few and too_many diagnostic arguments. Setting too_few = "align_start" or too_few = "debug" allows the pipeline to complete without crashing, populating unsupplied columns with NA or adding diagnostic columns to isolate offending records:
# Gracefully handle irregular delimiter counts across clinical rows
robust_split = raw_data %>%
separate_wider_delim(
cols = sample_code,
delim = "-",
names = c("site", "batch", "replicate"),
too_few = "align_start",
too_many = "drop"
)
4. Loss of Unobserved Factor Levels in Complete
When using complete() to generate all theoretical factor combinations, researchers occasionally find that unused levels of categorical factors disappear if the variable is stored as a raw character vector.
Fix: Convert categorical experimental variables to factors with explicitly declared levels before running complete(). The complete() function respects factor level sets and will construct rows for levels that have zero occurrences in the empirical dataset.
Free alternatives
While tidyr is the standard data reshaping package within the R tidyverse, researchers have multiple open-source alternatives across the R and Python programming ecosystems. Choosing the right tool depends on memory constraints, processing speed requirements, and institutional software stacks.
The comparative data table below summarizes the key open-source data tidying libraries across computational environments:
| Software / Package | Language / Ecosystem | Wide to Long Verb | Long to Wide Verb | Key Strengths | Primary Limitations |
|---|---|---|---|---|---|
| tidyr | R (Tidyverse) | pivot_longer() |
pivot_wider() |
Intuitive syntax; rich tidyselect integration; robust list rectangling. | Higher memory overhead on multi-gigabyte datasets than data.table. |
| data.table | R (High-Performance) | melt() |
dcast() |
Ultra-fast C-level execution; minimal memory footprint; in-place modification. | More compact, idiosyncratic syntax; steeper initial learning curve. |
| stats (Base R) | R (Core System) | reshape(direction='long') |
reshape(direction='wide') |
Zero external dependencies; built into every R installation; permanent stability. | Notoriously complex and counter-intuitive argument naming (idvar, timevar, varying). |
| reshape2 | R (Legacy Tidyverse) | melt() |
dcast() |
Historical predecessor to tidyr; widely documented in older literature. | Retired and superseded; lacks modern features like rectangling and tidyselect. |
| pandas | Python (PyData) | melt(), wide_to_long() |
pivot(), pivot_table() |
De facto standard in scientific Python; robust multi-index aggregation support. | MultiIndex structures can be difficult to flatten; higher memory usage. |
For research projects involving tabular datasets exceeding several million rows, data.table provides exceptional performance gains. However, for most scientific applications, tidyr’s expressive syntax, detailed error messaging, and seamless interoperability with ggplot2 and dplyr make it the most readable and maintainable choice for reproducible research workflows. Researchers conducting multi-language pipelines can explore wider concepts in our overview of data wrangling for research.
Frequently asked questions
What is the primary difference between tidyr and dplyr?
While both packages belong to the core R tidyverse, they serve distinct roles in the data manipulation lifecycle. The tidyr package focuses on the structural shape and layout of rectangular datasets, providing tools to pivot between wide and long representations, separate or unite text columns, handle missing observational rows, and rectangle nested lists. In contrast, the dplyr package focuses on data manipulation and transformation within an already tidy dataset, offering verbs to filter rows, select columns, compute summary statistics, create new variables with mutate, and join distinct tables.
Why were gather() and spread() replaced by pivot_longer() and pivot_wider()?
In tidyr version 1.0.0, Posit introduced pivot_longer() and pivot_wider() to replace the older gather() and spread() functions. Decades of user feedback revealed that the terminology of gathering and spreading was difficult to remember and counter-intuitive for researchers. Furthermore, gather() and spread() struggled with complex operations, such as pivoting multiple sets of interrelated columns simultaneously or handling compound column names containing multiple variables. The pivoting functions provide a consistent, expressive interface with robust support for tidyselect specifications, automated type conversion, and duplicate value handling.
How does tidyr handle implicit versus explicit missing values?
Explicit missing values are cells that physically contain an NA indicator within an existing row of data. Implicit missing values occur when an entire observation is missing from the dataset, such as an experimental subject who missed an annual checkup and therefore has no row recorded for that year. The tidyr package provides complete() to turn implicit missing values into explicit NA entries by calculating all combinations of specified experimental factors. Conversely, fill() imputes values by carrying preceding or subsequent values into NA cells, while drop_na() removes rows containing missing indicators.
When should researchers choose data.table instead of tidyr?
Researchers should consider using the data.table package when working with very large datasets (typically hundreds of megabytes or gigabytes in size) where computational execution time and memory footprint are critical bottlenecks. Because data.table::melt() and dcast() execute via compiled C code and perform in-place memory modifications, they process high-volume datasets significantly faster than tidyr. However, for small to medium-sized research datasets, tidyr offers clearer code readability, standardized integration with the tidyverse pipeline, and superior error reporting.
What is data rectangling in the tidyr package?
Data rectangling is the process of taking deeply nested, non-tabular data structures—such as JSON files, XML trees, or deeply nested lists returned by scientific web APIs—and converting them into a flat, rectangular data frame with tidy rows and columns. In tidyr, rectangling is accomplished through specialized verbs including unnest_longer() (which expands list-elements into rows), unnest_wider() (which expands list-elements into columns), and hoist() (which selectively extracts specific nested components from deeply structured records).
How does separate_wider_delim() improve on the older separate() function?
The separate_wider_delim() function was introduced in tidyr 1.3.0 to replace separate() with safer, more predictable behavior. While the legacy separate() function silently dropped extra pieces or padded missing values with NA without warning, separate_wider_delim() strictly validates that each row contains the expected number of delimited components. If an anomalous row contains too few or too many pieces, it halts execution and issues a diagnostic error unless the researcher explicitly configures the too_few or too_many arguments to dictate handling.
Sources
- CRAN Repository. Package tidyr: Tidy Messy Data (Version 1.3.2). https://cran.r-project.org/web/packages/tidyr/index.html (accessed 2026-11-15).
- Posit Software. Official tidyr Documentation and Function Reference. https://tidyr.tidyverse.org/ (accessed 2026-11-15).
- Posit Software. Pivoting Data Vignette: Reshaping with pivot_longer and pivot_wider. https://tidyr.tidyverse.org/articles/pivot.html (accessed 2026-11-15).
- Wickham, H. (2014). Tidy Data. Journal of Statistical Software, 59(10), 1-23. https://www.jstatsoft.org/article/view/v059i10 (accessed 2026-11-15).
- R Core Team. Base R stats::reshape Reference Manual. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/reshape.html (accessed 2026-11-15).
- data.table Development Team. Efficient Reshaping Using data.table melt and dcast. https://rdatatable.gitlab.io/data.table/reference/melt.data.table.html (accessed 2026-11-15).
- CRAN Repository. Package reshape2: Flexibly Reshape Data: A Reboot of the Reshape Package (Version 1.4.4). https://cran.r-project.org/web/packages/reshape2/index.html (accessed 2026-11-15).
Data: Statistical Software Data Reshaping and Tidying Tools Comparison
Comparative technical evaluation of primary tabular data reshaping and tidying libraries across R and Python environments, compiled from CRAN documentation, official package manuals, and peer-reviewed software references.
Period covered: 2024/2026. Coverage: Global.
| Package or Library | Language Environment | Wide to Long Function | Long to Wide Function | Execution Engine | Maintenance Status |
|---|---|---|---|---|---|
| tidyr | R (Tidyverse) | pivot_longer() | pivot_wider() | Vectorized C++ vctrs engine | Active (Posit PBC) |
| data.table | R (Comprehensive) | melt() | dcast() | In-place C-level memory mutation | Active (CRAN team) |
| stats (Base R) | R (Core) | reshape(direction='long') | reshape(direction='wide') | Interpreted Base R algorithmic loops | Active (R Core Team) |
| reshape2 | R (Historical) | melt() | dcast() | Standard R/C vectorized code | Retired / Superseded |
| pandas | Python (PyData) | melt() | pivot() / pivot_table() | Vectorized Cython and C/NumPy arrays | Active (NumFOCUS) |
Sources
- CRAN tidyr Package Repository (accessed 2026-11-15)
- Official tidyr Documentation (accessed 2026-11-15)
- CRAN data.table Package Index (accessed 2026-11-15)
- Base R stats::reshape Reference Manual (accessed 2026-11-15)
- CRAN reshape2 Package Index (accessed 2026-11-15)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







