Data Wrangling for Research: Tools and Steps

Data wrangling is the work of getting a raw dataset into a shape an analysis can use: fixing inconsistent text, deciding what a missing value means, and reshaping columns and rows. In research it matters because every one of those choices changes the numbers a paper reports.

What data wrangling is used for in research

Raw study data rarely arrives analysis-ready. Typical jobs are trimming stray whitespace, standardising case, converting data types, finding near-duplicate spellings of the same value, handling missing entries, and reshaping a wide spreadsheet into one row per observation. Each of these is a decision that belongs in the methods record, which is why wrangling connects directly to results reproducibility and to the longer-term work of data curation.

This guide covers three free tools that each represent a different working style: OpenRefine (point-and-click, interactive), pandas (Python code) and tidyr (R code). The point that most explainers skip is how each tool treats missing values and reshaping, because that is where silent errors enter a dataset.

Install or access (official sources only)

• OpenRefine: download from openrefine.org/download. The page offers an EXE installer or a ZIP file for Windows (the ZIP is described as including an embedded Java install), a DMG for Mac and a TAR.GZ for Linux.
• pandas: the official install page lists conda-forge and PyPI. With conda: conda install -c conda-forge pandas. With pip: pip install pandas. The pandas documentation recommends running pandas from a virtual environment.
• tidyr (R): the tidyr site says the easiest way to get it is to install the whole tidyverse: install.packages("tidyverse"). To install only tidyr: install.packages("tidyr").

Step-by-step: three core wrangling tasks

1. Clean text values in OpenRefine

1. Load your table into an OpenRefine project.
2. On the column menu choose Edit cells, then Transform. This opens an expressions window with a preview of the result before you apply it.
3. Use the Common transforms menu for trimming whitespace, changing case or converting data types.
4. To find near-duplicate text values, use clustering. OpenRefine always requires you to approve each suggested edit.
5. If a transform is wrong, use the Undo / Redo tab in the expressions window to return to an earlier state or reuse an expression you tried before.

2. Detect and handle missing values in pandas

1. Detect missing values with isna() or notna(). isna() also treats None as missing.
2. Decide per variable whether to drop or fill. dropna() drops rows or columns with missing data (pass axis=1 to drop columns). fillna() replaces NA values with non-NA data.
3. Record that decision in your script or analysis plan, because it changes the sample your statistics are computed on.

3. Reshape to tidy data with tidyr

The tidyr documentation defines tidy data by three rules: each variable is a column, each observation is a row, and each value is a cell. A spreadsheet with one column per year or per measurement occasion breaks the first rule. pivot_longer() makes a wide dataset longer, and pivot_wider() is its inverse. Both are documented in the vignette("pivot") article of the tidyr site.

Output and how to read it

The result of wrangling is a table plus a record of what was done to it. In pandas, missing entries are represented differently by data type: numpy.nan for NumPy dtypes (which coerces integer columns to float64 or object), NaT for datetime, timedelta and period types, and NA for the nullable types such as StringDtype, Int64Dtype, Float64Dtype and BooleanDtype. pandas does not use the nullable types by default; you need to specify the dtype explicitly.

Two behaviours change results without raising an error. Reductions such as a mean skip missing values by default; to include them pass skipna=False. And a count of rows before and after dropna() tells you how many observations your later analysis actually uses, so report both numbers.

Troubleshooting

A comparison with missing values does nothing

Comparisons with np.nan always return False, so df["x"] == np.nan will never select the missing rows. Test with isna() instead. With NA, comparisons propagate NA, and calling bool() on it raises a TypeError (“boolean value of NA is ambiguous”), which shows up when NA appears in an if condition.

An integer column turned into decimals

A NumPy-backed integer column that gains a missing value is coerced to float64 or object. Specify a nullable dtype explicitly if you need integers to stay integers.

Means or totals look too high or too low

Check whether missing values were skipped by default (skipna=True) and whether they should have been.

Free alternatives and how they compare

Tool	Working style	Documented strength
OpenRefine	Interactive desktop application	Cell transforms with a preview; clustering of near-duplicate text values that you approve one edit at a time; undo and redo.
pandas	Python code	Explicit missing-data methods (isna(), dropna(), fillna()) and per-dtype missing-value types.
tidyr	R code	Reshaping with pivot_longer() and pivot_wider() toward the three tidy-data rules.

If your team works in a licensed statistics package, see SPSS statistical analysis software and the SPSS vs R comparison. For tools aimed at curation workflows more broadly, see data curation tools, and for the metadata side of preparing data see metadata enrichment.

FAQ

What is data wrangling?

Data wrangling is the process of cleaning, structuring and reshaping raw data so it can be analysed. In practice it covers text clean-up, handling missing values and reshaping tables.

Is a scripted workflow more reproducible than a point-and-click one?

A script (pandas or R) can be re-run on the same input and reviewed line by line. In OpenRefine each transform is previewed and clustering edits are approved by the user, so keep a written record of the steps you applied. Either way, the cleaned dataset should be reported together with the steps that produced it.

Should I drop rows with missing values or fill them?

The tools only offer the mechanics (dropna(), fillna()). Which is appropriate depends on your study design and analysis plan, so state the choice and its effect on sample size in your methods.

Sources (official documentation)

• tidyr, “Tidy data” article: tidyr.tidyverse.org/articles/tidy-data.html (read 2026-10-29)
• tidyr home and install instructions: tidyr.tidyverse.org (read 2026-10-29)
• pandas user guide, “Working with missing data”: pandas.pydata.org/docs/user_guide/missing_data.html (read 2026-10-29)
• pandas installation: pandas.pydata.org/docs/getting_started/install.html (read 2026-10-29)
• OpenRefine manual, “Cell editing”: openrefine.org/docs/manual/cellediting (read 2026-10-29)
• OpenRefine downloads: openrefine.org/download (read 2026-10-29)