Written and maintained by CASRAI Editorial Board
Last updated
pandas duplicated() returns a boolean Series that marks which rows of a DataFrame (or which values of a Series) repeat an earlier or later one. It flags duplicates but does not remove them; removal is the job of drop_duplicates(). This guide is for researchers and analysts who need to find repeated records in a dataset, check why the flags are not what they expected, and decide what to do before deleting anything. Syntax and behaviour below follow the pandas 3.1.1 documentation read on 2026-11-07.
What pandas duplicated is used for in research
Repeated records appear in research data for ordinary reasons: a participant submitted a survey form twice, two exports were appended to one file, or a merge produced extra rows. duplicated() is the inspection step that comes before cleaning. Because it returns flags rather than a modified table, you can count, filter and review the flagged rows before deciding what counts as a true duplicate. See the wider data cleaning steps guide for where this fits in a cleaning workflow, and data profiling for checks that surface repeats earlier.
Install or access pandas (official source only)
The pandas installation page lists these commands:
pip install pandasconda install -c conda-forge pandas- To start a fresh environment:
conda create -c conda-forge -n name_of_my_env python pandas, thenconda activate name_of_my_env
Source: pandas installation guide. Supported Python versions are set out on the pandas Python support policy page, which is the page to check for your interpreter.
Syntax: DataFrame.duplicated(subset=None, keep=’first’)
| Argument | Default | What it does |
|---|---|---|
subset |
None |
A column label or a list of labels. Only these columns are compared. With the default, all columns are compared. |
keep |
'first' |
'first' marks duplicates as True except the first occurrence. 'last' marks duplicates as True except the last occurrence. False marks every member of a duplicated group as True. |
The return value is a boolean Series with one entry per row. The Series version, Series.duplicated(keep='first'), takes only keep. The documentation also lists Index.duplicated, Series.drop_duplicates and DataFrame.drop_duplicates as related methods.
Step-by-step: flag, count and inspect duplicate rows
The table below is an illustrative composite built for this guide, not real study data. Each row is one questionnaire submission.
import pandas as pddf = pd.DataFrame({ "participant_id": ["P01", "P02", "P02", "P03", "P03", "P01"], "visit": [1, 1, 1, 1, 1, 2], "score": [12, 15, 15, 9, 11, 14],})- Flag exact duplicate rows.
df.duplicated()compares all columns. Here only row 2 is True, because it repeats row 1 in every column. - Count them.
df.duplicated().sum()adds up the True values and gives 1. - Look at both members of each pair.
df[df.duplicated(keep=False)]uses the flags as a boolean mask and returns rows 1 and 2, so you can compare the whole group rather than only the later copy. - Choose the columns that define a repeat. Rows 3 and 4 share
participant_idandvisitbut have different scores, so they are not exact duplicates.df.duplicated(subset=["participant_id", "visit"])flags rows 2 and 4. That is a conflicting-record problem to resolve with the data owner, not a safe deletion. - Remove only after review.
df.drop_duplicates(subset=["participant_id", "visit"])keeps rows 0, 1, 3 and 5. Its parameters mirrorduplicated()(subset,keep) and addignore_index.
Output and how to read it: keep first, last and False
For the table above, the same data gives different flags depending on keep:
| Call | Flags for rows 0 to 5 | Reading |
|---|---|---|
df.duplicated() |
F, F, T, F, F, F | Row 2 is a later copy of row 1. |
df.duplicated(keep="last") |
F, T, F, F, F, F | Row 1 is flagged because a later copy exists. |
df.duplicated(keep=False) |
F, T, T, F, F, F | Every member of the group is flagged. |
df.duplicated(subset=["participant_id"]) |
F, F, T, F, T, T | Rows 2, 4 and 5 repeat an ID that appeared earlier. |
df.duplicated(subset=["participant_id", "visit"]) |
F, F, T, F, T, F | Row 5 is a different visit, so it is not a repeat. |
The official reference example (a table of instant noodle brands) shows the same pattern: by default only the second of two identical rows is True; keep="last" flips it to the first; keep=False marks both.
duplicated() versus drop_duplicates()
duplicated() answers “which rows repeat?” and returns a Series. DataFrame.drop_duplicates(subset=None, *, keep='first', inplace=..., ignore_index=False) returns a DataFrame with the repeats removed, using the same subset and keep semantics. In the pandas 3.1.1 documentation the inplace argument is marked deprecated since 3.1.0 and scheduled for removal in pandas 4.0, so assign the returned DataFrame instead. The documentation also notes that the index is ignored when finding duplicates and that the subset columns must be hashable, otherwise a TypeError is raised.
Troubleshooting: pandas duplicated not working
Nothing is flagged, or fewer rows than expected
The default compares all columns, so one differing value, for example a timestamp or a row ID column, makes two otherwise identical records distinct. Pass subset with the columns that define a repeat.
Only the second copy is flagged
That is the default keep='first'. Use keep=False to flag all members, or keep='last' to spare the final one.
I got a Series, not a table of duplicates
duplicated() returns booleans. Index the DataFrame with them: df[df.duplicated()].
Passing a column name versus a list
subset accepts a single label or an iterable of labels; a list such as ["participant_id", "visit"] is the usual form for several columns.
TypeError about unhashable values
Per the drop_duplicates documentation, the subset columns must be hashable. Columns holding lists or dicts need converting to a hashable form first.
Free alternatives in R and Python
| Tool | Call | Behaviour per its documentation |
|---|---|---|
| pandas | df.duplicated(subset, keep) |
Boolean Series, one entry per row. |
| Base R | duplicated(x, fromLast = FALSE) |
For a data frame, a logical vector with one element for each row. With fromLast = TRUE, the last of identical elements corresponds to FALSE. |
| dplyr | distinct(.data, ..., .keep_all = FALSE) |
Returns only the unique rows of a data frame; .keep_all = TRUE keeps all variables, retaining the first row for each combination. |
Related pandas guides on this site: pandas concat (a common source of repeated rows), pandas melt and pandas pivot. For unique values in NumPy arrays see numpy unique.
FAQ
What does pandas duplicated return?
A boolean Series, one value per row, where True marks a row treated as a duplicate according to keep.
How do I find duplicates based on multiple columns?
Pass a list to subset, for example df.duplicated(subset=["participant_id", "visit"]).
How do I see every copy, including the first?
Use df[df.duplicated(keep=False)].
Does duplicated() delete rows?
No. Use drop_duplicates() and assign the result.
Is duplicated pandas the same as Series.duplicated?
They share the keep argument. Series.duplicated checks values in one column and has no subset.
Sources
- pandas.DataFrame.duplicated, pandas documentation (3.1.1), read 2026-11-07.
- pandas.Series.duplicated, read 2026-11-07.
- pandas.DataFrame.drop_duplicates, read 2026-11-07.
- pandas installation guide, read 2026-11-07.
- R: Determine Duplicate Elements, R documentation, read 2026-11-07.
- dplyr distinct(), read 2026-11-07.
