pandas corr: Pairwise Correlation in Python

In empirical data analysis and scientific computing, evaluating bivariate relationships between variables is a fundamental analytical step. The pandas corr method, accessible as DataFrame.corr(), calculates pairwise correlation coefficients across numeric columns. Whether analyzing clinical biomarkers, survey instruments, or experimental trials, understanding how to configure correlation methods, handle missing observations, and interpret correlation matrices ensures reproducibility and prevents statistical errors.

What it is used for in research

Researchers routinely apply corr pandas operations during exploratory data analysis (EDA) and preliminary regression modeling. Primary research applications include:

• Multicollinearity screening: Identifying severe inter-variable dependence among predictors before fitting multivariable models, protecting regressions from inflated standard errors.
• Bivariate hypothesis screening: Quantifying linear or monotonic associations between experimental factors and outcomes across observational datasets.
• Methodological flexibility: Evaluating linear relationships with the Pearson correlation coefficient, monotonic relationships with Spearman rank correlation, or ordinal concordance via the Kendall Tau correlation coefficient.
• Pairwise missing-data handling: Computing coefficients using available observation pairs without discarding participant records that contain missing values in other columns.
• Exploratory matrix synthesis: Providing standardized association measures that complement unstandardized structures like the covariance matrix in multivariate workflows.

Install or access (official source only)

The pandas library is open-source software distributed through the Python Package Index (PyPI) and Conda. For research requiring statistical hypothesis testing, pairing pandas with scipy enables exact p-value and confidence interval calculations.

Install the packages using pip:

pip install pandas scipy

Alternatively, install them in an isolated Conda environment via conda-forge:

conda install -c conda-forge pandas scipy

To verify the installation in Python, check the package versions:

import pandas as pd
import scipy
print("pandas:", pd.__version__)
print("scipy:", scipy.__version__)

In pandas 2.0.0 and later releases, statistical methods enforce strict type discipline, making explicit parameter configuration necessary when working with mixed-type research DataFrames.

Step-by-step task (numbered, with menu names or commands exactly as in the current docs)

Follow these numbered steps to load research data, configure DataFrame.corr(), enforce type safety, and compute both correlation coefficients and inferential p-values.

Step 1: Construct the research DataFrame

Create a DataFrame containing continuous measurements, an ordinal stage score, and a non-numeric participant identifier:

import numpy as np
import pandas as pd
from scipy import stats

data = {
    "subject_id": ["S01", "S02", "S03", "S04", "S05", "S06"],
    "dose_mg": [10.0, 20.0, 30.0, 40.0, 50.0, 60.0],
    "biomarker": [1.4, 2.2, 2.9, np.nan, 4.3, 5.1],
    "response": [12.1, 15.4, 21.0, 26.8, 31.5, 38.2],
    "severity": [1, 2, 2, 3, 3, 4]
}

df = pd.DataFrame(data)

Step 2: Compute Pearson correlation with numeric_only=True

Calculate the standard Pearson correlation matrix across numeric columns while excluding non-numeric identifier fields:

# Compute pairwise Pearson correlation
pearson_mat = df.corr(method="pearson", numeric_only=True)
print(pearson_mat.round(4))

Setting numeric_only=True is required in pandas 2.0+ to prevent errors when DataFrames contain text strings or categorical fields.

Step 3: Calculate non-parametric rank correlations (Spearman and Kendall)

When analyzing ordinal variables or non-normally distributed data, select rank-based correlation methods:

# Spearman rank correlation
spearman_mat = df.corr(method="spearman", numeric_only=True)

# Kendall Tau correlation
kendall_mat = df.corr(method="kendall", numeric_only=True)

Spearman evaluates monotonic associations using ranked data, whereas Kendall Tau assesses concordant versus discordant pairs, offering greater reliability on small samples with tied ranks.

Step 4: Set minimum observation thresholds with min_periods

When datasets have missing values (NaN), enforce a minimum sample size per variable pair using min_periods:

# Require at least 5 complete observation pairs
filtered_corr = df.corr(method="pearson", min_periods=5, numeric_only=True)
print(filtered_corr.round(4))

Any column pair with fewer than 5 valid, non-null observations evaluates to NaN.

Step 5: Compute bivariate Series correlation and target-wise corrwith

To evaluate correlation between two specific Series, call Series.corr(). To compare all numeric columns against one specific outcome, use DataFrame.corrwith():

# Bivariate Series correlation
r_val = df["dose_mg"].corr(df["response"], method="pearson")

# Target-wise correlation across columns
target_corr = df.corrwith(df["response"], numeric_only=True)
print(target_corr.round(4))

Step 6: Compute p-values using SciPy

Because pandas computes point estimates without p-values, empirical studies combine pandas with SciPy to construct a companion significance matrix:

def get_pvalues(df_in):
    cols = df_in.columns
    p_mat = pd.DataFrame(np.zeros((len(cols), len(cols))), index=cols, columns=cols)
    for c1 in cols:
        for c2 in cols:
            if c1 == c2:
                p_mat.loc[c1, c2] = 0.0
            else:
                pair = df_in[[c1, c2]].dropna()
                if len(pair) >= 3:
                    p_mat.loc[c1, c2] = stats.pearsonr(pair[c1], pair[c2]).pvalue
                else:
                    p_mat.loc[c1, c2] = np.nan
    return p_mat

p_values = get_pvalues(df.select_dtypes(include=[np.number]))
print(p_values.round(5))

Output and how to read it

The output of DataFrame.corr() is a symmetric two-dimensional matrix. To interpret the statistical structure, review the principles in how to read a correlation matrix.

Key properties of the output matrix include:

• Principal diagonal: Every variable correlated with itself equals 1.0, forming a diagonal of ones.
• Symmetry: The matrix is symmetric across the diagonal, meaning r(A, B) = r(B, A).
• Value bounds: Coefficients range from -1.0 (perfect inverse linear relationship) to +1.0 (perfect direct linear relationship), with 0.0 reflecting no linear relationship.
• NaN entries: Occur when a column has zero variance (constant values), when overlapping non-null cases fall below min_periods, or when missing data leaves zero complete pairs.

Troubleshooting

When running correlation analyses in pandas, common operational challenges include:

• ValueError: could not convert string to float:
  • Cause: In pandas 2.0.0+, numeric_only defaults to False. If non-numeric columns exist, pandas attempts calculations on text data, triggering a ValueError.
  • Resolution: Pass numeric_only=True inside corr() or filter numeric columns with df.select_dtypes(include='number').corr().
• Unexpected NaN values across entire rows or columns:
  • Cause: Zero variance in a variable (all values identical) causes division by zero, or missing cases fall below the min_periods threshold.
  • Resolution: Check variances using df.var(numeric_only=True) and verify valid counts with df.notna().sum().
• Spurious correlations from minimal overlapping data:
  • Cause: Pairwise deletion computes coefficients using whichever rows have mutual data, sometimes yielding deceptive values from only two or three shared rows.
  • Resolution: Set min_periods=10 (or your study’s required minimum sample size) to suppress under-powered pairs to NaN.
• Performance slowdown with custom callables:
  • Cause: Supplying a custom callable to method executes unvectorized Python loops across all column pairs.
  • Resolution: Use the built-in vectorized options ('pearson', 'spearman', 'kendall') for wide datasets.

Free alternatives (table)

The following open-source statistical tools offer alternative approaches for calculating correlation matrices in scientific research:

Tool / Package	Environment	Supported Methods	P-Value Calculation	Missing Data Strategy	Primary Research Advantage
pandas (DataFrame.corr)	Python	Pearson, Spearman, Kendall, callable	No (point estimate only)	Pairwise deletion (configurable via min_periods)	Fast tabular integration in Python data workflows
SciPy (scipy.stats)	Python	Pearson, Spearman, Kendall	Yes (exact p-values and confidence intervals via PearsonRResult)	Requires manual masking or nan_policy	Rigorous inferential statistics and hypothesis testing
Pingouin (pairwise_corr)	Python	Pearson, Spearman, Kendall, robust biweight	Yes (p-values, 95% CI, Bayes factors, and power)	Pairwise or listwise deletion	Comprehensive publication-ready statistical tables
R (base::cor and Hmisc::rcorr)	R	Pearson, Spearman, Kendall	Yes (in Hmisc::rcorr; point estimates in base cor)	Configurable via use=”pairwise.complete.obs”	Biostatistical standard with integrated significance matrices
Julia (StatsBase.cor)	Julia	Pearson, Spearman, Kendall	No (requires HypothesisTests.jl)	Pairwise filtering with skipmissing	High-performance compiled execution across large arrays

FAQ

How do I run corr pandas with non-numeric text columns?

To run correlation on a DataFrame with non-numeric columns, pass numeric_only=True: df.corr(numeric_only=True). In pandas 2.0.0 and later, omitting this parameter when non-numeric columns exist raises a ValueError. Alternatively, select numeric features first with df.select_dtypes(include='number').corr().

Why does pandas corr not include p-values or significance levels?

The DataFrame.corr() function is designed for exploratory data analysis and point estimation. It does not compute sampling distributions or test statistics. For research requiring hypothesis testing, calculate p-values using scipy.stats.pearsonr or the pingouin.pairwise_corr() library.

How does pandas corr handle missing values compared to dropna()?

By default, DataFrame.corr() applies pairwise deletion: for any two columns, rows are excluded only if either of those two specific variables is null, preserving data for other pairs. In contrast, calling df.dropna().corr() applies listwise deletion, removing any row with a missing value across any column.

What is the difference between Pearson, Spearman, and Kendall methods in pandas corr?

The Pearson method evaluates linear associations between continuous, normally distributed variables. The Spearman method evaluates monotonic relationships using ranked values, accommodating skewed continuous or ordinal data. The Kendall method measures ordinal concordance based on concordant versus discordant pairs, offering greater robustness on small samples with tied ranks.

Sources (official docs)

• pandas Development Team: pandas.DataFrame.corr API Reference: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html (verified 2026-11-11).
• pandas Development Team: pandas.Series.corr and pandas.DataFrame.corrwith API Reference: https://pandas.pydata.org/docs/reference/api/pandas.Series.corr.html (verified 2026-11-11).
• pandas Development Team: pandas 2.0.0 Release Notes (numeric_only parameter default change): https://pandas.pydata.org/docs/whatsnew/v2.0.0.html (verified 2026-11-11).
• SciPy Community: scipy.stats.pearsonr API Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html (verified 2026-11-11).