What Is Data Profiling? A Practical Guide for Research Datasets

Data profiling is the process of examining a dataset to understand its structure, contents and quality before you analyze or clean it. IBM describes it as “the process of reviewing and cleansing data to better understand how it’s structured”. In practice, for a researcher, it means producing a column-by-column summary: how many values are present, how many are missing, how many distinct values there are, and what the ranges look like. This guide shows the documented command for four tools (pandas, DuckDB, ydata-profiling and OpenRefine) and how to read what each one returns.

What data profiling is used for in research

Profiling is a diagnosis step: you look before you fix. It answers questions such as “is this column really numeric?”, “how many rows are blank?” and “do these category labels repeat with small spelling differences?” Results feed directly into a research data quality framework, into decisions about missing values (see diagnosing missing data) and into the metadata you document for the dataset.

IBM’s overview groups profiling into three kinds of work:

• Structure discovery: checking format consistency and patterns, for example a phone number that is missing a digit.
• Content discovery: finding row-level errors.
• Relationship discovery: understanding how datasets connect, which starts with metadata analysis.

The tools below mostly cover structure and content discovery for a single table, which is the usual case for a research spreadsheet or CSV export.

Install or access

• pandas: DataFrame.describe is part of the pandas library; see the pandas documentation for installation.
• DuckDB: SUMMARIZE is a DuckDB SQL statement; see the DuckDB documentation for how to access DuckDB.
• ydata-profiling: pip install -U ydata-profiling
• OpenRefine: facets are built into OpenRefine; see the OpenRefine manual.

Step-by-step: profile a table four ways

1. Summarize every column with pandas. The documented signature is DataFrame.describe(percentiles=None, include=None, exclude=None). By default it summarizes numeric columns only (and falls back to object or categorical columns if there are no numeric ones). To cover every column, call df.describe(include='all').
2. Summarize a table in SQL with DuckDB. Run SUMMARIZE tbl; for a table, or SUMMARIZE SELECT * FROM tbl; for the result of a query.
3. Generate a full report with ydata-profiling. After installing, run:

   import pandas as pd
   from ydata_profiling import ProfileReport
   df = pd.read_csv('data.csv')
   profile = ProfileReport(df, title="Profiling Report")

   The project describes the report as including “a wide range of statistics and visualizations, providing a holistic view of your data”, shareable as an HTML file or as a widget in a Jupyter Notebook, and notes it excels at “identification of missing data, duplicate entries and outliers”.

4. Inspect values interactively with OpenRefine facets. Open a column’s dropdown, choose Facet, then Text facet to list identical text values with counts, or Numeric facet to see a histogram you can filter by dragging the minimum and maximum. For empty cells, use the null, empty or blank facet in the column’s Facet menu.

Output and how to read it

pandas describe()

For numeric columns the result has count, mean, std, min, max and percentiles (25 and 75 by default; the 50 percentile is the same as the median). For object columns it returns count, unique, top and freq, where top is the most common value and freq is its count. With include='all', statistics that do not apply to a column are shown as NaN.

DuckDB SUMMARIZE

The output has one row per column with these fields: column_name, column_type, min, max, approx_unique, avg, std, q25, q50, q75, count and null_percentage. Note the name approx_unique: the distinct count is an approximation, so do not treat it as an exact number of categories. null_percentage is the quickest way to spot a mostly empty column.

OpenRefine facets

A text facet groups identical text and shows counts, sortable alphabetically or by count. Matching is exact: the manual says it “does no guessing about typos or near-matches”, so values that differ at all are listed as separate choices. The Cluster feature is the one that finds near-duplicates.

Troubleshooting

• describe() shows only some columns. The default include=None summarizes numeric columns only. Pass include='all' to see the rest.
• The top value looks arbitrary. When several values tie for most common, the pandas documentation says the one chosen for top is arbitrary. Check the text facet in OpenRefine or a group-by count if ties matter.
• NaN appears in a describe() table. With include='all', NaN marks a statistic that does not apply to that column type; it does not by itself mean missing data.
• Blank cells show as 0 in a numeric facet. The OpenRefine manual notes that blank, non-numeric or error values can show as “0” in the numeric facet. Check blanks with the blank facet before trusting the histogram.

Free alternatives

Tool	Command or entry point	What it returns
pandas	df.describe(include='all')	count, mean, std, min, max and percentiles for numeric columns; count, unique, top and freq for object columns
DuckDB	SUMMARIZE tbl;	One row per column with type, min, max, approx_unique, avg, std, quartiles, count and null_percentage
ydata-profiling	ProfileReport(df, title="Profiling Report")	A report of statistics and visualizations, shareable as HTML or a Jupyter widget
OpenRefine	Column dropdown, Facet, Text facet or Numeric facet	Value counts, histograms and blank-value facets, with Cluster for near-duplicates

Profiling only diagnoses. Once you know what is wrong, cleaning is a separate step; OpenRefine’s facets and Cluster feature are a natural follow-on for text fixes, and numeric summaries pair with descriptive statistics in Stata or the notes on descriptive versus inferential statistics. If your data comes from REDCap, see how missing-data codes export before profiling.

Frequently asked questions

What is data profiling?

It is the process of examining a dataset to understand its structure, content and quality, usually by computing per-column summaries such as counts, missing values, distinct values and ranges.

What is the difference between profiling and cleaning?

Profiling reports what is in the data; cleaning changes it. Profile first so that each cleaning step responds to a problem you have actually seen.

Which tool should a researcher start with?

If your data is already in Python, df.describe(include='all') is a single line. If it sits in a CSV or database you query with SQL, SUMMARIZE returns one row per column. If you prefer a point-and-click interface, use OpenRefine facets.

Sources

Official documentation, checked 2026-10-29.

• pandas documentation, DataFrame.describe (checked 2026-10-29).
• DuckDB documentation, Summarize guide (checked 2026-10-29).
• ydata-profiling documentation, docs.profiling.ydata.ai and its installation page (checked 2026-10-29).
• OpenRefine manual, Facets (checked 2026-10-29).
• IBM, What is data profiling?, used for the definition and the three types of profiling (checked 2026-10-29).