Written and maintained by CASRAI Editorial Board
Published · Last updated
On this page
In R, pivoting data from wide to long format reshapes a data frame so that multiple measurement columns are collapsed into key-value pairs, producing a tidy structure where each row represents a single observation and each column represents a single variable. The modern standard function for this operation is pivot_longer() from the tidyr package (part of the core tidyverse). It supersedes older reshaping commands like gather() by providing a more expressive interface for handling complex column names, multi-variable prefixes, and in-flight data type conversions.
What it is used for in research
In scientific research, clinical trials, and epidemiological studies, raw data is frequently recorded or exported in wide format. For instance, an electronic data capture system or laboratory assay instrument often outputs a separate column for each measurement timepoint (such as baseline, week 4, and week 12 scores). While wide tables are easy for human data entry, most downstream statistical analysis tools and graphics packages in R require tidy (long) data.
Researchers routinely pivot data long for three primary purposes:
- Statistical Modeling: Repeated-measures analysis of variance (ANOVA), generalized estimating equations (GEE), and linear mixed-effects models (such as those fitted with
lme4::lmerornlme::lme) require one row per observation, with an explicit variable indexing the repeated time or condition factor. - Data Visualization: The
ggplot2graphics library relies on tidy aesthetics. Plotting longitudinal trajectories or faceted boxplots across experimental conditions requires grouping variables to be stored in dedicated columns rather than dispersed across column headers. - Harmonized Data Pipelines: Integrating data across multiple collection sites as part of a broader data wrangling workflow requires converting varied schemas into a canonical structure that clearly delineates distinct types of variables.
Unlike simply transposing matrices and data frames in R, which inverts the entire matrix orientation without regard to variable semantics, pivot_longer() selectively lengthens target measurement columns while preserving subject identifiers and experimental covariates.
How to install or access it
The pivot_longer() function is maintained in the open-source tidyr package, published on the Comprehensive R Archive Network (CRAN) under the MIT license by Posit Software, PBC. It is compatible with R versions 4.1.0 and higher.
To install tidyr from official CRAN mirrors, run the standard package installer in the R console:
install.packages("tidyr")
Alternatively, if you use the complete tidyverse suite for research analysis, you can install the complete meta-package:
install.packages("tidyverse")
Once installed, load the library into your active R script or interactive session:
library(tidyr)
To confirm that your installed version supports all modern pivoting features, inspect the package version:
packageVersion("tidyr")
Step-by-step instructions
Follow these numbered steps to reshape a wide research dataset into tidy long format using tidyr::pivot_longer().
-
Inspect the wide dataset and identify invariant key columns.
Determine which columns should remain intact as observation identifiers (such as subject IDs, trial arms, or demographic factors) and which columns contain measurement values spread across headers. Consider this representative clinical trial dataset:
library(tidyr) # Sample wide research dataset: 3 participants with repeated biomarker measurements trial_wide <- data.frame( patient_id = c("PT-101", "PT-102", "PT-103"), arm = c("Control", "Treatment", "Treatment"), biomarker_m0 = c(14.2, 16.5, 12.8), biomarker_m3 = c(14.0, 14.1, 11.2), biomarker_m6 = c(14.5, 12.3, 10.1) ) print(trial_wide) -
Execute a basic pivot using
cols,names_to, andvalues_to.Specify the columns to pivot using tidyselect helpers, name the new key column that will hold the old header strings, and name the new column that will hold the measurement values:
trial_long <- pivot_longer( data = trial_wide, cols = starts_with("biomarker_"), names_to = "timepoint", values_to = "biomarker_value" ) print(trial_long)You can also use negative column selection (e.g.,
cols = -c(patient_id, arm)) to pivot all columns except the designated identifier fields. -
Strip header prefixes with
names_prefix.To eliminate redundant text from the resulting key column, supply a regular expression to
names_prefix. For instance, to remove the leading"biomarker_"text and isolate the time labels:trial_clean_time <- pivot_longer( data = trial_wide, cols = starts_with("biomarker_"), names_to = "month", names_prefix = "biomarker_m", values_to = "biomarker_value" ) print(trial_clean_time) -
Decompose multi-variable column headers using
names_sepornames_pattern.When wide column names encode multiple experimental factors (for example, assay name and timepoint, such as
crp_wk0andcrp_wk4), split them into distinct tidy columns in one call:# Multi-variable wide dataset complex_wide <- data.frame( patient_id = c("PT-101", "PT-102"), crp_wk0 = c(1.2, 3.4), crp_wk4 = c(1.1, 2.1), il6_wk0 = c(5.4, 7.8), il6_wk4 = c(5.2, 5.9) ) complex_long <- pivot_longer( data = complex_wide, cols = -patient_id, names_to = c("biomarker", "visit"), names_sep = "_", values_to = "concentration" ) print(complex_long) -
Enforce strict type conversions using
names_transformandvalues_transform.Column headers are initially parsed as character strings. To convert numerical indices (like numeric months or dose amounts) directly into integers or factors during pivoting, supply a list of transform functions:
trial_typed <- pivot_longer( data = trial_wide, cols = starts_with("biomarker_"), names_to = "month_num", names_prefix = "biomarker_m", names_transform = list(month_num = as.integer), values_to = "biomarker_value" ) # Confirms month_num is integer rather than character str(trial_typed$month_num) -
Manage missing observations with
values_drop_na.If participants missed specific visits, their wide rows contain
NAcells. Setvalues_drop_na = TRUEto drop missing rows from the long output, or retain them (values_drop_na = FALSE, default) to maintain explicit record completeness compliant with FAIR data principles:# Drops rows where biomarker_value is NA trial_complete_cases <- pivot_longer( data = trial_wide, cols = starts_with("biomarker_"), names_to = "timepoint", values_to = "biomarker_value", values_drop_na = TRUE )
Output and how to read it
The output of pivot_longer() is a restructured data frame (or tibble) with fewer columns and more rows than the input.
To verify the transformation, compare the dimensions and column classes before and after pivoting:
# Input dimensions: 3 rows by 5 columns
dim(trial_wide)
# Output dimensions: 9 rows by 4 columns (3 patients * 3 timepoints)
dim(trial_long)
When inspecting the printed output of trial_long, look for the following characteristics:
- Repeating Keys: Each
patient_idandarmnow appears 3 times, once for each corresponding measurement row. - Key Column (
timepoint): Contains the names of the original columns (e.g.,"biomarker_m0","biomarker_m3","biomarker_m6"). - Value Column (
biomarker_value): Contains the numerical readings that were previously dispersed across horizontal headers. - Row Ordering: By default,
cols_vary = "fastest"arranges the output rows so that all timepoints for a single patient appear together consecutively, which matches the required format for longitudinal modeling.
Troubleshooting
When reshaping complex research datasets, you may encounter specific errors or unexpected outputs. Here is how to diagnose and resolve them:
- Error: Can’t combine <character> and <double>:
Cause: The columns passed to
colscontain conflicting data types. For example, if one timepoint column contains numeric values while another contains text strings (such as"<0.01"or"missing"), R cannot merge them into a single atomic vector.Resolution: Inspect column classes using
sapply(df, class). Harmonize data types prior to pivoting (e.g., replacing text flags withNAusingdplyr::na_if()), or force coercion during pivoting by addingvalues_transform = list(values_to = as.character). - Error: ‘names_to’ contains 2 elements, but ‘names_sep’ / ‘names_pattern’ was not supplied:
Cause: Supplying multiple output column names in
names_to = c("var1", "var2")without telling the function how to split the incoming header strings.Resolution: Add
names_sep = "_"for simple character delimiters, or provide regular expression capture groups innames_pattern = "(.*)_(.*)". - Unintended Row Multiplication or Missing Identifiers:
Cause: The columns not selected in
colsdo not uniquely identify each observational unit, causing R to duplicate rows or group multiple measurements together unexpectedly.Resolution: Ensure the non-pivoted columns form a composite primary key. If your dataset lacks a unique subject or run identifier, generate one prior to pivoting using
trial_wide$row_id <- seq_len(nrow(trial_wide)). - Columns Pivoting Unexpectedly:
Cause: Loose pattern matching in tidyselect helpers (e.g.,
starts_with("m")matching bothmonth_1andmedical_record_number).Resolution: Use strict character vectors
cols = c("month_1", "month_2")or precise negative selectionscols = -c(medical_record_number, site_id).
Free alternatives
While tidyr::pivot_longer() is the standard in the tidyverse ecosystem, several other free and open-source alternatives exist for reshaping data in R and related research computing environments.
| Tool / Function | Environment / Package | Syntax Approach | Memory & Performance | Best Research Application |
|---|---|---|---|---|
tidyr::pivot_longer() |
R (tidyr / tidyverse) |
Declarative tidyselect syntax with integrated splitting (names_sep) and casting |
Fast C++ execution via vctrs; optimized for standard in-memory analysis |
General scientific analysis, tidy data workflows, and direct input preparation for ggplot2. |
stats::reshape() |
Base R (Built-in stats) |
Argument-driven (varying, timevar, idvar, direction = "long") |
Minimal overhead; zero external dependencies | Base R scripts, lightweight server automation, and package development minimizing external imports. |
data.table::melt() |
R (data.table) |
Vector arguments (id.vars, measure.vars, variable.name) |
Extremely fast in-place memory management; handles tens of millions of rows | High-throughput genomics, electronic health record registries, and memory-constrained compute clusters. |
pandas.melt() |
Python (pandas) |
DataFrame method (id_vars, value_vars, var_name, value_name) |
C/Cython vectorized execution within the Python scientific stack | Cross-language research projects and workflows manipulating columns in pandas. |
FAQ
What is the difference between pivot_longer() and the older gather() function?
pivot_longer() was introduced in tidyr version 1.0.0 to supersede gather(). While both convert wide data to long format, pivot_longer() provides a cleaner and more consistent syntax that aligns with pivot_wider(), supports modern tidyselect helpers directly, and allows multi-variable column headers to be parsed into separate columns using names_sep and names_pattern without requiring external string manipulation.
How do I keep specific identifier columns from being pivoted?
To exclude identifier columns from pivoting, use the negative selection operator - in the cols argument. For example, cols = -c(patient_id, treatment_group) instructs R to retain patient_id and treatment_group as intact identifying columns while pivoting all remaining columns into key-value pairs.
Can pivot_longer() split column names into multiple new variables?
Yes. By specifying multiple names in the names_to argument (such as names_to = c("measurement", "time")) and providing either names_sep = "_" or a regex pattern in names_pattern, pivot_longer() splits compound column headers and places the resulting parts into distinct tidy variables simultaneously.
Does pivoting long in R change the underlying data values?
No. Pivoting is strictly a structural reshaping operation that rearranges rows and columns. It does not alter, recompute, or aggregate your recorded data values unless you explicitly request a data type conversion via values_transform or omit missing values using values_drop_na = TRUE.
When should researchers use long format instead of wide format?
Researchers should use long format whenever they need to fit repeated-measures regression models (such as linear mixed models or GEEs), visualize grouped data in ggplot2, or conduct group-level summaries across repeated experimental intervals. Wide format is primarily useful for initial manual data entry and tabular report presentation.
Sources
- Posit Software, PBC. tidyr: Pivot data from wide to long (pivot_longer reference). URL: https://tidyr.tidyverse.org/reference/pivot_longer.html (checked 2026-11-10).
- Comprehensive R Archive Network (CRAN). Package ‘tidyr’: Tidy Messy Data (Version 1.3.2, Authors: Hadley Wickham, Davis Vaughan, Maximilian Girlich). URL: https://cran.r-project.org/web/packages/tidyr/index.html (checked 2026-11-10).
- Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. R for Data Science (2e), Chapter 5: Data tidying. URL: https://r4ds.hadley.nz/data-tidy.html (checked 2026-11-10).







