Python for Exploratory Data Analysis: A pandas Walkthrough

Exploratory data analysis (EDA) in Python usually means loading a dataset into a pandas DataFrame, checking its structure and missing values, summarising each variable, and looking at relationships and plots before any formal test. This walkthrough uses only the methods documented in the pandas manual, and spends most of its space on the part tutorials skip: what each output actually tells you, and where the defaults quietly leave things out.

What Python EDA is used for in research

EDA is the look-around step between data collection and modelling. In a research project it answers practical questions: did the file load with the columns and types you expected, how much is missing and where, what do the distributions look like, and which variables move together. The answers decide whether you proceed to data cleaning, a pandas and SciPy analysis, or back to the data source. EDA is a way of looking, not a test: it generates questions to check later, and it should be recorded in a script so a reader can rerun it.

Install or access (official source only)

The pandas installation page gives two routes. Both are best used inside a dedicated environment, so the versions you used for the analysis can be recorded and recreated.

• pip: install into a virtual environment (for example one made with Python’s standard-library venv), then run pip install pandas.
• conda: the pandas documentation recommends Miniforge. Create an environment with conda create -c conda-forge -n name_of_my_env python pandas, then conda activate name_of_my_env.
• Plotting extras: pip install "pandas[plot, output-formatting]" installs the optional plotting and output-formatting dependencies. The 10 minutes to pandas tutorial imports matplotlib.pyplot for plots, so make sure matplotlib is available.

Step-by-step: a first EDA pass in pandas

These steps use the commands shown in the current pandas documentation. Replace foo.csv with your own file.

1. Load the data. import pandas as pd, then df = pd.read_csv("foo.csv").
2. Look at the first and last rows. df.head() shows the first five rows by default; df.tail() shows the last rows. Check that the columns line up with your codebook.
3. Check structure. df.info() prints a concise summary: the index dtype and columns, non-null counts and memory usage.
4. Count missing values. df.isna() returns a boolean frame the same size as df. Summing it by column (df.isna().sum()) gives a count per variable, because True counts as 1. dropna() and fillna() exist for removing or filling missing values, but in EDA the first job is only to see the pattern.
5. Summarise the variables. df.describe() gives a quick statistical summary. Add include='all' to cover non-numeric columns as well.
6. Count categories. df["column"].value_counts() returns the count of each unique value; normalize=True returns relative frequencies instead.
7. Check relationships. df.corr(numeric_only=True) returns a correlation matrix; the default method is Pearson, and method also accepts 'kendall' and 'spearman'.
8. Plot. import matplotlib.pyplot as plt, then df.plot(). Outside Jupyter, call plt.show() to display the figure or plt.savefig("figure.png") to write it to a file. For a colour-scaled view of a correlation matrix, see heatmaps in seaborn.

Output and how to read it

Reading info()

The table has the columns #, Column, Non-Null Count and Dtype, followed by a dtypes: line and a memory usage: line. Compare each Non-Null Count with the number of rows: any column with a smaller count has missing values. A column you expected to be numeric but listed as object usually holds text mixed in with the numbers. Two settings matter: show_counts is, by default, shown only when the frame is smaller than the max_info_rows and max_info_columns options, so on a very wide or long dataset the counts can be absent unless you ask for them. And memory_usage='deep' performs a true memory calculation, at a computational cost.

Reading describe()

For numeric columns, describe() reports count, mean, std, min, 25%, 50%, 75% and max; the 50% row is the median. Three behaviours are easy to miss:

• By default only numeric columns are summarised. A text or categorical column silently disappears from the output (it is described instead only when there are no numeric columns at all).
• With include='all', all columns are shown, and a statistic that does not apply to a column is filled with NaN. Object columns report count, unique, top and freq instead; when several values tie for top, the one chosen is arbitrary.
• NaN values are excluded from the calculations, so count is the number of non-missing values, not the number of rows. Compare it with info().

Reading corr() when data are missing

The correlation is computed from pairwise complete observations: each pair of columns uses the rows where both have a value. So one cell of the matrix can rest on a different number of rows than another cell, and a matrix is not a single sample. min_periods sets the minimum number of observations required per pair; below it the cell is NaN. From pandas 2.0.0 the numeric_only default is False, so pass numeric_only=True when your frame contains text columns. For interpretation, see how to read a correlation matrix, and for why missingness patterns matter, diagnosing MCAR, MAR and MNAR.

A missing-value check that can mislead

The pandas documentation for isna() states that NA values such as None or NaN map to True, and that everything else maps to False: characters such as empty strings '' or numpy.inf are not considered NA. A column of blanks or placeholder codes can therefore show zero missing values. Look at value_counts() for the columns that matter before trusting the count.

Troubleshooting

• A column is missing from describe(). It is not numeric. Use include='all', or check its dtype in info().
• info() shows no Non-Null Count. Counts are shown by default only for frames smaller than the max_info_rows and max_info_columns options; pass show_counts=True.
• The correlation matrix contains NaN. A pair had fewer observations than min_periods, or a column is constant or entirely missing.
• Correlation fails on a mixed frame. Pass numeric_only=True so non-numeric columns are left out.
• No plot appears outside a notebook. Call matplotlib.pyplot.show, or save the figure with savefig.
• Memory figure looks too small. Use info(memory_usage='deep') for a real calculation.

Free alternatives

Option	What it is	Source
pandas with matplotlib	The route above: DataFrame methods for loading, summaries and missing-value checks, with df.plot() drawing through matplotlib.	pandas documentation
jamovi	A free and open statistical spreadsheet with a point-and-click interface, for those who prefer not to write code. See the jamovi entry.	jamovi.org (checked 2026-10-29)

FAQ

Is pandas enough for exploratory data analysis?

For a first pass, yes: loading, structure, missing values, summaries, category counts and correlations are all pandas methods, and plotting goes through matplotlib. Specialised needs such as clustering are covered in separate guides.

What is the difference between info() and describe()?

info() describes the structure of the frame: columns, dtypes, non-null counts and memory. describe() describes the values: count, mean, spread and quartiles for numeric columns.

How do I find missing values in pandas?

Use df.isna() for a boolean mask and sum it per column, and compare info() non-null counts with the row count. Remember that empty strings are not treated as missing by isna().

Should I drop or fill missing values during EDA?

EDA is for seeing the pattern. dropna() and fillna() change the data, so apply them as a deliberate, documented cleaning step afterwards.

Sources (official documentation, checked 2026-10-29)

• pandas, Installation: https://pandas.pydata.org/docs/getting_started/install.html
• pandas, 10 minutes to pandas: https://pandas.pydata.org/docs/user_guide/10min.html
• pandas, DataFrame.info: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.info.html
• pandas, DataFrame.describe: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html
• pandas, DataFrame.corr: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html
• pandas, DataFrame.isna: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isna.html
• pandas, Series.value_counts: https://pandas.pydata.org/docs/reference/api/pandas.Series.value_counts.html
• jamovi: https://www.jamovi.org/