Skip to main content
v2026.11,986 entries · CC-BY 4.0

str.split in Pandas: Syntax, Arguments, and Research Examples

Master str.split in pandas to segment string columns with delimiters or regex, handle missing values, and expand results into tidy research DataFrames.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

In pandas, the Series.str.split() method splits strings in a Series or Index around a designated delimiter, regular expression, or whitespace, returning either lists of substrings or a multi-column DataFrame. By setting the expand=True parameter, researchers can unpack delimited survey responses, biological identifiers, or compound geographic variables into structured columns for immediate downstream statistical modeling.

Ask CASRAI · free to tryQuestion about this guide? Ask CASRAI answers with linked sources. 2 free questions a day, no account.

Ask your first question freeSubscribe

What it is used for in research

Empirical scientific workflows frequently encounter datasets where multiple distinct measurements, hierarchical classifications, or observational metadata are concatenated within a single textual field. In public health registries, patient identifiers often encode clinical site codes, enrollment cohorts, and subject numbers separated by hyphens (such as SITE04-COHORT2-SUBJ109). In bioinformatics and molecular biology, FASTA headers, GenBank accession records, and genomic variant identifiers routinely concatenate chromosome numbers, base-pair intervals, and reference alleles using colons and underscores. Similarly, social science surveys administered through platforms like Qualtrics or REDCap frequently export multi-select questionnaire items as semicolon-delimited character strings within a single column.

Performing statistical analysis or data visualization requires disaggregating these composite strings into discrete, tidy variables. In base Python, processing such columns typically involves iterating through rows with list comprehensions or applying user-defined functions via apply(). However, row-by-row iteration in Python incurs substantial interpreter overhead and loses vectorised execution benefits. The pandas Series.str.split() method provides vectorised string operations implemented directly on top of NumPy arrays or Apache Arrow string buffers. Understanding your underlying pandas string data types ensures that memory allocation remains efficient when handling millions of textual observation records.

Beyond elementary string segmentation, Series.str.split() integrates directly with pandas reshaping mechanics. By toggling the expand argument, analysts can transform composite text into new tabular dimensions, or combine the output with Series.explode() to normalize one-to-many relationships into tidy observation rows. This disaggregation is foundational for subsequent data imputation, categorical grouping, and longitudinal regression analysis across diverse research domains.

How to install or access it

The str.split() method is a core component of the pandas library and does not require third-party add-ons. Pandas can be installed and updated using either the Python Package Index (PyPI) or the Conda package management system through the official conda-forge channel.

To install pandas in a virtual environment via PyPI, run the following command in your terminal:

pip install pandas

For scientific computing environments utilizing Anaconda or Miniconda, install pandas from the community-maintained conda-forge channel:

conda install -c conda-forge pandas

For optimal computational performance with large research text corpora, installing PyArrow enables Arrow-backed string storage, which accelerates string splitting operations and significantly reduces memory overhead:

pip install pyarrow

After installation, verify that the environment is properly configured by launching an interactive Python session, importing the library, and checking the installed version:

import pandas as pd
print(pd.__version__)

Official pandas documentation supports Python versions 3.9 and higher for modern releases (pandas 2.0 and later). If you are assembling a fresh data processing pipeline, review our guide on creating a DataFrame in Python to ensure your raw input structures are formatted appropriately prior to string parsing.

Step-by-step task

The following step-by-step tutorial demonstrates how to parse, expand, and structure compound string variables from a research dataset. In this scenario, an environmental epidemiology study tracks air quality monitoring stations where each station record contains a compound identifier (e.g., REGION-STATION_ID-YEAR) and a delimited string of observed chemical pollutants (e.g., PM2.5;NO2;SO2;O3).

Step 1: Construct the research sample DataFrame. Initialize the dataset containing compound identifiers, pollutant readings, and observation dates:

import pandas as pd
import numpy as np

data = {
    "station_code": ["NE-ST01-2023", "MW-ST04-2023", "SO-ST12-2024", "WE-ST09-2024", np.nan],
    "measured_pollutants": ["PM2.5;NO2;SO2", "O3;PM10", "PM2.5;O3;CO;NO2", "SO2", "PM2.5;NO2"],
    "site_coordinates": ["42.3601 N, 71.0589 W", "41.8781 N, 87.6298 W", "29.7604 N, 95.3698 W", "37.7749 N, 122.4194 W", "40.7128 N, 74.0060 W"]
}

df = pd.DataFrame(data)
print(df)

Step 2: Execute basic delimiter splitting without expansion. By default, calling .str.split() on a Series without arguments splits on whitespace. To split on a specific delimiter such as a hyphen or semicolon, pass the character string to the pat parameter:

# Splitting pollutants into Python lists
pollutant_lists = df["measured_pollutants"].str.split(pat=";")
print(pollutant_lists)

The resulting Series retains the original index and contains Python list objects. When an element in the Series is NaN, pandas preserves the missing value without raising an exception.

Step 3: Expand split elements into separate DataFrame columns. To convert the delimited elements into structured tabular columns, specify expand=True. This returns a DataFrame where each token occupies its own column:

# Expand station code into separate columns
station_parts = df["station_code"].str.split(pat="-", expand=True)
print(station_parts)

The returned DataFrame contains integer column labels (0, 1, 2). You can rename these columns and integrate them back into your main analysis DataFrame using standard assignment, or by adding columns to a pandas DataFrame with descriptive variable names:

df[["region", "station_id", "sample_year"]] = df["station_code"].str.split(pat="-", expand=True)
print(df[["region", "station_id", "sample_year"]])

Step 4: Constrain split count with the n parameter. In cases where a string contains multiple delimiter instances but you only wish to extract the leading segment or preserve trailing text, specify the n parameter. Setting n=1 restricts the operation to a single split:

# Split only on the first hyphen
partial_split = df["station_code"].str.split(pat="-", n=1, expand=True)
partial_split.columns = ["region_code", "remaining_identifier"]
print(partial_split)

Step 5: Perform right-to-left splitting using str.rsplit(). When parsing file paths, hierarchical taxons, or URLs, splitting from the end of the string is often necessary. The Series.str.rsplit() method operates identically to str.split() but scans from right to left when constrained by n:

# Split once from the right to isolate the year
right_split = df["station_code"].str.rsplit(pat="-", n=1, expand=True)
right_split.columns = ["station_prefix", "record_year"]
print(right_split)

Step 6: Split using regular expression patterns. When input strings contain inconsistent or multiple delimiters, regular expressions provide flexible segmentation. Pass a regex pattern to pat and ensure regex=True:

# Split coordinates on comma or whitespace
coord_split = df["site_coordinates"].str.split(pat=r"[, ]+", expand=True, regex=True)
print(coord_split)

Step 7: Reshape delimited strings to long-format observations with explode(). For repeated-measures or hierarchical models where each multi-select entry should represent an individual observation row, combine str.split(expand=False) with the Series.explode() method:

# Reshape DataFrame to long format: one row per measured pollutant
tidy_df = df[["station_code", "measured_pollutants"]].copy()
tidy_df["pollutant"] = tidy_df["measured_pollutants"].str.split(pat=";")
tidy_df = tidy_df.explode("pollutant").reset_index(drop=True)
print(tidy_df.head(8))

Output and how to read it

The structure and data type of the output returned by Series.str.split() depend directly on the expand argument, the caller type, and the consistency of the delimiter count across rows.

When expand=False (the default setting), the output is a pandas Series with an object data type. Each cell in the Series contains a standard Python list of strings. For example, the string "PM2.5;NO2;SO2" yields ['PM2.5', 'NO2', 'SO2']. If the input contains a missing value (such as np.nan or None), the corresponding output cell remains NaN. Individual elements within these lists can be accessed vectorially using string indexing syntax, such as df['pollutant_list'].str[0] to retrieve the first token or df['pollutant_list'].str[-1] to retrieve the final token.

When expand=True, the output is converted into a full pandas DataFrame. The columns of this DataFrame are indexed by sequential integers starting at zero (0, 1, 2, ...), corresponding to the position of each split segment. The total number of columns equals the maximum number of splits found in any single row plus one. If a row contains fewer delimiters than the row with the maximum count, trailing columns for that row are populated with None (or np.nan when using standard object storage, or pd.NA under Arrow string storage). Researchers should be prepared to verify column completeness before executing mathematical operations.

When applied to a pandas Index or MultiIndex object, calling index.str.split(expand=True) produces a MultiIndex. This behavior provides a rapid pathway for converting delimited row index labels into multi-level hierarchical indices for grouped aggregations. The data table below compares string splitting parameters, return types, and null behavior across scientific computing libraries to help researchers select the appropriate method for their analytical pipelines.

Troubleshooting

When working with heterogeneous or real-world research datasets, string splitting operations can trigger unexpected exceptions or silent data quality issues. The following solutions address the most common obstacles.

1. AttributeError: Can only use .str accessor with string values.

This error occurs when attempting to call .str.split() on a Series that pandas stores as an integer, float, or datetime data type. Even if the column contains values that look like identifiers (e.g., numeric patient codes or dates), pandas must recognize the column as string data. To resolve this, explicitly cast the column to a string type prior to splitting:

df["code_str"] = df["numeric_code"].astype(str)
df["code_str"].str.split("-", expand=True)

In pandas 2.0+, using .astype("string") or .astype("string[pyarrow]") provides stricter type validation and avoids unexpected conversions of missing values into the literal string "nan".

2. Unexpected splits caused by regex interpretation.

In pandas, the regex parameter defaults to None. When regex=None, pandas checks the length of the string passed to pat: single-character strings are treated as literal delimiters, while multi-character strings are interpreted as regular expressions. If your delimiter is a literal character with special meaning in regex syntax—such as a period (.), pipe (|), or plus sign (+)—passing a multi-character string like "+=" or passing regex=True without escaping can cause incorrect splitting. For example, splitting on an unescaped dot pat="." matches every character rather than a literal period. Always specify regex=False when splitting on literal multi-character strings, or escape special characters using re.escape():

# Splitting on literal period safely
df["filename"].str.split(pat=r".", expand=True, regex=True)
# Or equivalently:
df["filename"].str.split(pat=".", expand=True, regex=False)

3. Sparse columns resulting from unequal delimiter counts.

When splitting free-text survey responses or user-entered notes, rows often contain variable numbers of delimiters. Setting expand=True creates columns up to the maximum observed delimiter count, leaving trailing columns filled with None or NaN. Downstream statistical functions may fail when encountering these uneven distributions. Researchers should inspect column null counts using df.isna().sum() and employ systematic strategies for dropping missing values with dropna or filtering incomplete observations prior to analysis.

4. High memory consumption with large text Series.

By default, pandas stores string columns in NumPy arrays using the object data type. An object Series contains pointers to individual Python string objects scattered across memory. When splitting millions of records, generating lists of strings increases memory fragmentation. Converting string columns to PyArrow-backed storage (StringDtype(storage="pyarrow")) stores text in contiguous memory buffers, reducing memory footprint by up to 70% and accelerating string operations.

Free alternatives

While pandas is the standard data manipulation library in Python, several free and open-source alternatives offer string splitting capabilities tailored to different computational scales, memory constraints, and programming environments.

The Python standard library provides built-in str.split() and re.split() methods. These require no external installations and are ideal for lightweight scripts or single-string transformations. However, they lack tabular vectorization and require manual list comprehensions to apply across datasets, which becomes cumbersome when aligning results with multi-column DataFrames.

Polars is a blazingly fast DataFrame library written in Rust that utilizes Apache Arrow as its native memory format. In our detailed Polars versus pandas comparison, Polars demonstrates significant speed advantages on large datasets due to multi-threaded execution and strict typing. Polars provides Expr.str.split() to return a Series of lists, and Expr.str.split_exact() to parse strings into fixed-width structs that can be unnested into separate columns with zero-copy memory overhead.

For distributed big data processing, Apache PySpark provides the pyspark.sql.functions.split() function. PySpark distributes string operations across compute clusters, making it suitable for datasets that exceed single-machine memory capacity. In the R ecosystem, the stringr package (part of the tidyverse) provides str_split() for returning lists and str_split_fixed() for returning character matrices, while base R offers the built-in strsplit() function.

The following table summarizes the key operational characteristics of these free, open-source string splitting tools:

Tool / Environment Primary Method / Function Default Delimiter Return Container (Default) Rectangular Expansion Mechanism Execution Model
Python pandas Series.str.split() Whitespace Series of lists expand=True parameter Single-threaded (vectorized C/NumPy/Arrow)
Python Standard Library str.split() Whitespace List of strings Manual unpacking / list comprehension Single-threaded interpreter
Python Polars Expr.str.split() Required delimiter Series of lists str.split_exact() into struct Multi-threaded parallel (Rust / Arrow)
Apache PySpark functions.split() Regex pattern (required) Column of array type Array indexing or inline() Distributed cluster computing
R stringr (tidyverse) str_split() Regex pattern (required) List of character vectors str_split_fixed() to matrix Vectorized C / C++ (stringi)
R base strsplit() Regex pattern (required) List of character vectors do.call(rbind, ...) wrapper Vectorized C internals

Frequently asked questions

How do I split a pandas column into multiple columns?

To split a string column into separate columns, call .str.split() with the argument expand=True. This returns a DataFrame where each token becomes an individual column. You can directly assign the resulting DataFrame to a list of new column names in your original DataFrame, for example: df[['first_name', 'last_name']] = df['full_name'].str.split(' ', expand=True).

What is the difference between str.split() and str.rsplit() in pandas?

The str.split() method splits strings starting from the left (beginning) of each string, whereas str.rsplit() splits starting from the right (end). When returning all splits (the default n=-1), both methods produce identical outputs. However, when limiting the number of splits using the n parameter, str.rsplit() preserves the leading portion of the string and isolates the specified number of segments from the right.

How does pandas str.split() handle missing values (NaN or None)?

Pandas propagates missing values gracefully during string splitting operations. If a row contains np.nan, None, or pd.NA, calling .str.split() does not raise an exception. When expand=False, the resulting cell retains the missing value indicator. When expand=True, the resulting row contains missing values across all generated columns.

When should I use str.split() versus str.partition() in pandas?

Use str.split() when you need to divide a string around all occurrences of a delimiter, when the number of delimiters varies across rows, or when using regular expressions. Use str.partition() when you specifically need to split a string into exactly three parts based on the first occurrence of a separator: the text before the separator, the separator itself, and the text after the separator.

How can I turn the list of split strings into separate rows instead of columns?

To convert the elements of a split string into individual rows, first call .str.split(expand=False) to create a Series of lists. Next, apply the Series.explode() method to that column. This duplicates the remaining row data for each element in the list, producing a normalized long-format DataFrame ideal for relational database storage and repeated-measures analysis.

Sources

  • pandas Development Team (2024). pandas.Series.str.split API Reference. PyData. URL: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.split.html (accessed: 2026-11-16)
  • pandas Development Team (2024). Working with Text Data: pandas User Guide. PyData. URL: https://pandas.pydata.org/docs/user_guide/text.html (accessed: 2026-11-16)
  • pandas Development Team (2024). pandas.Series.str.rsplit API Reference. PyData. URL: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.rsplit.html (accessed: 2026-11-16)
  • Python Software Foundation (2024). Built-in Types: str.split. Python 3.12 Documentation. URL: https://docs.python.org/3/library/stdtypes.html#str.split (accessed: 2026-11-16)
  • Polars Development Team (2024). polars.Expr.str.split Documentation. Polars API Reference. URL: https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.str.split.html (accessed: 2026-11-16)
  • Apache Software Foundation (2024). pyspark.sql.functions.split API Reference. Apache Spark Python API. URL: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.split.html (accessed: 2026-11-16)
  • R Core Team (2024). strsplit: Split the Elements of a Character Vector. R Manual. URL: https://stat.ethz.ch/R-manual/R-devel/library/base/html/strsplit.html (accessed: 2026-11-16)
  • Wickham, H., & Posit Software (2024). stringr::str_split: Split up a string into pieces. Tidyverse Reference. URL: https://stringr.tidyverse.org/reference/str_split.html (accessed: 2026-11-16)

Data: String Splitting Syntax, Return Types, and Delimiter Mechanics Across Data Science Engines

An original comparative matrix detailing parameter configurations, return types, regex handling, and null propagation for string splitting functions across Python, R, and distributed scientific data processing libraries.

String Splitting Syntax, Return Types, and Delimiter Mechanics Across Data Science Engines
Tool or LibraryFunction or MethodDefault DelimiterReturn Type DefaultTable Expansion OptionRegex SupportNull Propagation
pandasSeries.str.split()Whitespace (contiguous)Series of listsexpand=True parameterSupported (regex flag or pattern)Preserves NaN as missing
Python Standard Librarystr.split()Whitespace (contiguous)List of stringsCustom list unpack or zipNot supported (requires re.split)Raises AttributeError on None
PolarsExpr.str.split()Required delimiterSeries of listsstr.split_exact() into structSupported via str.split()Preserves null as null
Apache PySparkpyspark.sql.functions.split()Regex pattern (required)Column of array typeinline() or array indexingNative (regex pattern by default)Returns null on null input
R stringr (tidyverse)stringr::str_split()Regex pattern (required)List of character vectorsstr_split_fixed() returns matrixNative (regex pattern by default)Preserves NA
R basebase::strsplit()Regex pattern (required)List of character vectorsdo.call(rbind, ...) wrapperNative (regex by default, fixed=TRUE)Preserves NA
DuckDBstr_split() / string_split()Required delimiter stringVARCHAR[] arrayUNNEST or struct unpackingSupported via regexp_split_to_arrayPreserves NULL

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about str.split in Pandas: Syntax, Arguments, and Research Examples

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →