pandas drop_duplicates(): Syntax, subset, keep and Examples

Who this is for: researchers and analysts who have a pandas DataFrame with repeated records and need to remove them in a way they can defend and reproduce. DataFrame.drop_duplicates() returns a DataFrame with duplicate rows removed; by default it compares every column and keeps the first occurrence of each row.

Signature in the pandas 3.1.1 reference (read 2026-11-07): DataFrame.drop_duplicates(subset=None, *, keep='first', inplace=<no_default>, ignore_index=False). The same call is often searched as “drop duplicates pandas”; it is one method. To only find duplicates without removing them, see pandas duplicated().

What it is used for in research

Research datasets pick up repeated rows through merged exports, re-run instrument uploads, repeated survey submissions and appended files (see pandas concat). Removing them is one step of data cleaning, and it changes your sample size, so it belongs in the documented, scripted part of the analysis rather than in a manual edit. Counting duplicates first is also a natural output of data profiling.

Install or access (official source only)

pandas is installed from its official install page (pandas.pydata.org). The pandas documentation gives pip install pandas and, for conda users, conda install -c conda-forge pandas. Check what you have with import pandas as pd; print(pd.__version__). The examples below were run on pandas 3.0.5; the reference page for 3.1.1 documents the same signature plus the inplace deprecation described below.

Step-by-step: remove duplicate rows

The example dataset has seven rows. Row 2 repeats row 1 exactly; sample S03 appears twice with different readings; sample S05 differs from S04 only by letter case and a trailing space.

import numpy as npimport pandas as pddf = pd.DataFrame({    "sample_id": ["S01", "S02", "S02", "S03", "S03", "S04", "S05"],    "site": ["North", "North", "North", "South", "South", "East", "east "],    "reading": [4.1, 3.8, 3.8, 5.0, 5.2, np.nan, np.nan],})

1. Audit first. Count what you are about to remove and record it: n_dup = df.duplicated().sum() (here 1). duplicated() is covered in its own guide.
2. Drop exact duplicate rows. clean = df.drop_duplicates(). The result has 6 rows: row 2 is removed, because it matches row 1 in every column.
3. Assign the result. drop_duplicates() returns a new DataFrame; the original df is unchanged unless you reassign it.
4. Log the change. print(len(df), len(clean)) gives 7 6. Keep that line in your script output so the exclusion is reproducible.

Drop duplicates based on one or two columns (subset)

Pass subset a column label or a list of labels; only those columns are compared, and the default is all columns. “Pandas drop duplicates based on two columns” is therefore df.drop_duplicates(subset=["sample_id", "site"]), and one column is df.drop_duplicates(subset=["sample_id"]). In the example, subset=["sample_id"] keeps rows 0, 1, 3, 5 and 6, so the first S03 reading (5.0) survives and 5.2 is discarded. An unknown column label in subset raises KeyError.

Choose which record survives (keep)

keep accepts three values: 'first' (default) drops duplicates except the first occurrence, 'last' drops them except the last occurrence, and False drops every row that has a duplicate.

• df.drop_duplicates(subset=["sample_id"], keep="last") keeps rows 0, 2, 4, 5, 6: S03 now keeps 5.2.
• df.drop_duplicates(subset=["sample_id"], keep=False) keeps rows 0, 5, 6: both S02 rows and both S03 rows are removed. Use this when a repeated ID means the record cannot be trusted.

Choosing a survivor by rule. “First” means first in current row order, which may be arbitrary after a merge. To keep, for example, the highest reading per sample, sort first: df.sort_values("reading", ascending=False).drop_duplicates("sample_id"). In the example this keeps S03 with 5.2. This sort-then-drop pattern is an editorial technique, not a separate pandas feature; the behaviour comes from keep='first'.

Reset the row labels (ignore_index)

ignore_index=True relabels the result 0, 1, …, n-1 instead of leaving gaps where rows were removed: df.drop_duplicates(subset=["sample_id"], ignore_index=True).

Do not use inplace=True

The reference marks inplace as deprecated since pandas 3.1.0, to be removed in pandas 4.0. Write df = df.drop_duplicates(...) instead. In pandas 3.0.5, inplace=True returns None, a common source of “my DataFrame became None” errors when the call is assigned.

Duplicate index labels

The reference states that duplicates in the index are not considered. To drop repeated index labels, use a boolean mask on the index, as the reference shows: indexed[~indexed.index.duplicated(keep="first")]. Run on a frame with index ["a", "a", "b"], this keeps the first a row and the b row. The pandas user guide on duplicate labels (pandas.pydata.org/docs/user_guide/duplicates.html) also describes resolving repeated labels by aggregation, for example groupby(level=0).mean(), when you want to combine rather than discard them.

Near-duplicates: normalise first

Matching is exact. "East" and "east " are different values, so S04 and S05 above are not treated as duplicates of each other. If they should match, build a normalised helper column and drop on it:

df["site_key"] = df["site"].str.strip().str.lower()df.drop_duplicates(subset=["site_key"])

Document the normalisation rule alongside the result; it is a judgement about what counts as the same record, not something pandas decides for you.

Output and how to read it

• The return value is a DataFrame holding the surviving rows with their original index labels (gaps are normal unless you set ignore_index=True).
• Row order is preserved.
• Missing values: in the tested example, rows with NaN in the same columns compare as duplicates (two rows of NaN, 1 collapsed to one). Check this against your own data before relying on it for blank-heavy survey records.
• Compare len() before and after, and report the number removed and the rule used (all columns, which subset, which keep).

Troubleshooting

Symptom	Likely cause and fix
Rows that look identical are not removed	Hidden differences: case, trailing whitespace or a different float value. Normalise into a helper column and use subset.
Result is None	inplace=True returns None. Drop inplace and assign the result.
KeyError naming a column	A label in subset is misspelt or absent. Check df.columns.
Wrong record kept	Default is keep='first' in current row order. Sort first, or use keep='last'.
Repeated index labels remain	The method ignores the index. Use ~df.index.duplicated().
TypeError about unhashable values	The reference notes that columns in subset must be hashable. Convert list-valued cells to tuples or strings first.

Free alternatives

Tool	Call	What its documentation says
pandas (Python)	DataFrame.drop_duplicates(subset, keep=...)	keep is ‘first’, ‘last’ or False.
Polars (Python)	DataFrame.unique(subset=None, *, keep='any', maintain_order=False)	keep can be ‘any’ (no guarantee which row is kept), ‘none’, ‘first’ or ‘last’; maintain_order=True keeps original order at extra cost.
dplyr (R)	distinct(.data, ..., .keep_all = FALSE)	“Keep only unique/distinct rows from a data frame”; with .keep_all = TRUE it retains all columns, keeping the first row for each combination.

Note the Polars default differs from pandas: pandas keeps the first row by default, while Polars 'any' gives no such guarantee.

FAQ

How do I drop duplicates in pandas based on multiple columns?

Pass a list to subset: df.drop_duplicates(subset=["col_a", "col_b"]). Rows are duplicates only when all listed columns match.

Which duplicate does pandas keep?

The first, by default (keep='first'). Use keep='last' for the last, or keep=False to remove every row that has a duplicate.

How do I find duplicates in a column before dropping them?

Use duplicated() to get a boolean mask, for example df[df.duplicated(subset=["sample_id"], keep=False)] to view every repeated record. See the pandas duplicated() guide.

How do I drop duplicate index values?

df[~df.index.duplicated(keep="first")]; drop_duplicates() does not consider the index.

Does drop_duplicates change my original DataFrame?

No. It returns a new DataFrame. The inplace option is deprecated since pandas 3.1.0.

Is dropping duplicates always correct?

No. Repeated measurements can be legitimate. Decide from the study design which columns identify a unique record, and report the rule and the count removed.

Sources (official documentation)

• pandas, pandas.DataFrame.drop_duplicates (3.1.1), read 2026-11-07.
• pandas, Duplicate Labels, read 2026-11-07.
• Polars, DataFrame.unique, read 2026-11-07.
• dplyr, distinct(), read 2026-11-07.
• Code outputs described above were produced by running the examples on pandas 3.0.5.