What Is Parquet? The Columnar File Format Explained

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It is used to store large tables, such as exported research datasets, so that software can read only the columns it needs. This page covers the data file format; “parquet” the wooden flooring pattern is unrelated.

After reading you should be able to explain how a Parquet file is organised, open a .parquet file, convert a CSV file to Parquet, and decide when to use it. For a format-by-format decision against plain text, see Parquet vs CSV; for array-oriented alternatives, see HDF5 vs Zarr vs Parquet.

What Parquet is used for in research

Parquet stores tabular data by column rather than by row, with “high performance compression and encoding schemes” and support for complex data in bulk (Apache Parquet overview, read 2026-11-07). It can be read from Python and SQL tools such as pandas and DuckDB. It is a storage format, not an analysis program: you open it with a library or query engine.

The format specification is maintained in the Apache parquet-format repository, which holds only the specification and the parquet.thrift metadata definitions. parquet-java contains the Java libraries, and further implementations exist in C++, Go, Rust, DuckDB and ClickHouse. The Apache documentation notes that implementations do not all support the same features, so a file written by one tool is not guaranteed to use only features every other reader supports.

What is inside a Parquet file

This is the part most introductions skip, and it explains why Parquet can be fast to read (Apache Parquet file format documentation, read 2026-11-07).

• A Parquet file begins and ends with the 4-byte magic number PAR1.
• A table of N columns is split into M row groups. Each row group holds one column chunk for each column.
• The file metadata is written in the footer, after the data, “to allow for single pass writing”.
• Readers are expected to read the file metadata first to find the column chunks they are interested in.

The practical consequence: because the footer tells a reader where each column chunk lives, a query that needs two columns of a fifty-column table can go straight to those chunks. A row-oriented text file offers no such index.

Benefits of Parquet files

• Column orientation. Data is organised by column, which suits queries that touch a few columns.
• Compression and encoding. The format provides compression and encoding schemes designed for efficient storage.
• Selective reads. The footer metadata lets readers locate only the column chunks of interest.
• Broad implementation support. Libraries and engines exist across several languages (see the table below).

How to open a .parquet file

A Parquet file is not meant to be opened in a text editor. Use a reader. Two documented routes follow.

Open with DuckDB

The DuckDB documentation (read 2026-11-07) gives these statements:

SELECT * FROM 'test.parquet';

If the file does not have a .parquet extension, use the reader function:

SELECT * FROM read_parquet('test.parq');

Open with pandas

The signature in the pandas 3.1.1 documentation (read 2026-11-07) is pandas.read_parquet(path, engine=<no_default>, columns=None, storage_options=None, dtype_backend=<no_default>, filesystem=None, filters=None, to_pandas_kwargs=None, **kwargs). The function requires the pyarrow library, which is the engine used by default; the 'fastparquet' and 'auto' engine options are deprecated since 3.1.0. Pass columns= to read a subset of columns. Reading from S3 URLs requires s3fs.

How to convert a CSV file to Parquet

The DuckDB documentation shows this example for converting a CSV file:

1. Install DuckDB from its official site and start a session.
2. Run: COPY 'test.csv' TO 'result-uncompressed.parquet' (FORMAT parquet, COMPRESSION uncompressed);
3. To convert the result of a query instead, run: COPY (SELECT * FROM tbl) TO 'result-snappy.parquet' (FORMAT parquet);

Alternatively, load the CSV into a pandas DataFrame and write it with pandas’ Parquet writer, DataFrame.to_parquet; as with reading, this needs a Parquet engine such as pyarrow installed.

Output and how to read it

To check what a file contains without loading all rows, DuckDB documents these statements:

DESCRIBE SELECT * FROM 'test.parquet';
SELECT * FROM parquet_schema('test.parquet');
SELECT * FROM parquet_metadata('test.parquet');

DESCRIBE lists the columns a query returns, parquet_schema shows the file’s schema, and parquet_metadata exposes the footer metadata described above.

Troubleshooting

• pandas raises an import error. pandas.read_parquet requires the pyarrow library; install it in the same Python environment.
• The file has no .parquet extension. Query it with read_parquet('file') in DuckDB rather than the bare quoted-filename form.
• An S3 path fails in pandas. S3 URLs require s3fs.
• A tool reads a file another tool wrote but not all features work. Implementations do not all support the same features, so test with the reader your collaborators will use.
• Passing engine='fastparquet' or 'auto' warns. Both are deprecated since pandas 3.1.0.

Free tools that read Parquet

Tool	Role	Source
DuckDB	SQL queries and CSV conversion with COPY	DuckDB Parquet documentation
pandas with pyarrow	Read into a DataFrame with read_parquet	pandas 3.1.1 documentation
parquet-java	Java libraries for the format	Apache Parquet overview
C++, Go and Rust implementations; ClickHouse	Other implementations listed by the Apache project	Apache Parquet overview

For other Python-based research tooling, see our guides to NetworkX and the OpenAlex API; for planning file formats within a project, see the data management plan entry.

FAQ

What is Parquet?

An open source, column-oriented data file format designed for efficient data storage and retrieval, maintained by the Apache project.

What is Parquet material?

That is a different meaning of the word: parquet flooring, a wooden floor pattern. This page is about the data file format.

Is a Parquet file human readable?

Not in a text editor. The file starts and ends with the PAR1 magic number and keeps its metadata in a footer for readers to parse, so open it with a reader such as DuckDB or pandas.

How can I open a .parquet file?

Query it with SELECT * FROM 'test.parquet'; in DuckDB, or call pandas.read_parquet with pyarrow installed.

How do I convert a CSV file to Parquet?

In DuckDB, use COPY 'test.csv' TO 'result.parquet' (FORMAT parquet);, following the documented COPY form shown above.

What are the benefits of Parquet files?

Column orientation, compression and encoding schemes, selective reading of needed column chunks through footer metadata, and wide implementation support.

Sources

• Apache Parquet, Overview: parquet.apache.org/docs/overview (read 2026-11-07).
• Apache Parquet, File Format: parquet.apache.org/docs/file-format (read 2026-11-07).
• pandas 3.1.1, pandas.read_parquet: pandas.pydata.org (read 2026-11-07).
• DuckDB, Parquet overview: duckdb.org (read 2026-11-07).