Hugging Face Datasets: Install, Load, Stream, Create

Who this is for: a researcher who wants to pull a published dataset into Python, stream one too large to download, or publish their own, and to record it well enough that the work can be reproduced. Everything below follows the official Hugging Face Datasets documentation (v4.8.4), read 2026-11-07.

What Hugging Face Datasets is used for in research

Hugging Face Datasets is a Python library, datasets, that loads data from the Hugging Face Hub, from local files or from memory into one object type with a uniform interface for slicing, mapping, filtering and export. The Hub (huggingface.co) hosts community-shared datasets as git-based repositories with revisions, covering audio, vision and natural-language data. See the Hugging Face Hub entry for the platform concept and Hugging Face Papers for the paper-linking side of the site.

Hugging Face vs Kaggle

The Hugging Face documentation does not compare itself with Kaggle, so this page does not characterise Kaggle. What the documentation does establish is what Hugging Face offers: a Hub of community datasets stored as versioned repositories, plus a library that loads them with load_dataset.

Install or access (official source only)

The library is tested on Python 3.10+. The documentation recommends a virtual environment, created with python -m venv .env.

• pip install datasets
• Audio support: pip install datasets
• Image support: pip install datasets[vision]
• conda: conda install -c huggingface -c conda-forge datasets
• From source: git clone https://github.com/huggingface/datasets.git, then cd datasets and pip install -e .

Deep-learning frameworks are installed separately (for example pip install torch or pip install tensorflow). To check the install, run python -c "from datasets import load_dataset; print(load_dataset('rajpurkar/squad', split='train')[0])"; it prints the first SQuAD training example with the keys answers, context, id, question and title.

How to use a Hugging Face dataset: step by step

1. Load it. from datasets import load_dataset, then dataset = load_dataset("nyu-mll/glue", "mrpc", split="train"). The arguments are the dataset name, an optional configuration and a split.
2. Pin the version. Pass revision= with a tag, branch name or commit hash so a rerun gets the same data.
3. Take a subset if needed. Splits can be sliced: split="train[10:20]", split="train[:10%]" or combined with split="train+test".
4. Transform it. dataset.map(fn, batched=True) applies a function in batches; other methods include rename_column, filter, select_columns and cast_column("audio", Audio(sampling_rate=16000)). For multi-shard datasets num_proc=8 speeds up preparation.
5. Hand it to a framework. set_format or with_format(type="torch") returns framework tensors.
6. Export if you need to. to_csv, to_json, to_parquet, to_sql, to_pandas, to_polars and to_dict.

In the quickstart’s NLP example, mapping a tokenizer over the dataset with batched=True adds the columns input_ids, token_type_ids and attention_mask, and the label column is renamed to labels.

Choose a loading route

Situation	Call (as in the documentation)	Result
Dataset on the Hub	load_dataset("nyu-mll/glue", "mrpc", split="train")	Downloads and caches a Dataset
Too large to download	load_dataset('HuggingFaceFW/fineweb', split='train', streaming=True)	An IterableDataset, usable immediately; the documentation gives the English fineweb split as 45 TB
Local CSV	load_dataset("csv", data_files="my_file.csv")	Dataset from file
Local JSON	load_dataset("json", data_files="my_file.json")	JSON Lines recommended; nested JSON needs field="data"
Local Parquet	load_dataset("parquet", data_files={'train':'train.parquet','test':'test.parquet'})	Dataset with named splits
Already in Python	Dataset.from_dict, from_list, from_generator, from_pandas	In-memory Dataset

Other supported local formats include arrow, hdf5 and webdataset, and SQL data via Dataset.from_sql. By default, all data lands in the train split unless file names mention train, test or validation; data_files={"train":"train.csv","test":"test.csv"} maps files to splits explicitly.

Streaming a large dataset

An IterableDataset is not for random access. shuffle(seed=42, buffer_size=10_000) shuffles within a buffer (default 1,000); take(n) and skip(n) select by position, and the documentation notes they prevent later calls to shuffle, so shuffle before splitting. Call set_epoch(epoch) to reshuffle each epoch, and Dataset.to_iterable_dataset(num_shards=64) to convert an in-memory Dataset. Streamed results can be saved as Parquet with push_to_hub.

Record the dataset so the work is reproducible

This is the part a how-to usually skips. For a methods section, note the dataset identifier, the configuration and split, and the revision you pinned. Read the dataset card: the documentation describes cards as helping users find a dataset and understand how to use it responsibly, with metadata fields for license, language and task categories, so check the license field before reuse. For the structure of good dataset documentation see datasheet for datasets and model cards and system cards; for the wider practice see reproducibility and sharing a reproducible conda environment.

How do I create a dataset for machine learning?

Through the website

1. From your profile choose New Dataset, give it a name and choose public or private (private is visible only to you or your organisation).
2. Open the Files and versions tab and choose Add file. Supported text, audio and image extensions include .csv, .mp3 and .jpg.
3. Choose Create Dataset Card to create the README.md; the metadata interface has license, language and task-category fields, and an Import dataset card template option.

The documentation recommends compressing .csv, .json, .jsonl and .txt files (.zip or .gz). Text extensions are not tracked by Git LFS by default, and files over 10MB will not be committed or uploaded unless compressed.

From Python

1. pip install huggingface_hub
2. huggingface-cli login
3. dataset.push_to_hub("username/processed_demo"); add private=True when creating the repository for the first time. Use num_proc=8 for parallel upload of streamed data.

Output and how to read it

Indexing a Dataset returns a dictionary of column names to values, as in the SQuAD check above. An IterableDataset does not support random access, so inspect it with take(n).

Troubleshooting

Symptom	Cause and fix (per the documentation)
A huge, unintended download	Without data_files, all files of a repository load (the documentation cites C4 at about 13TB). Pass data_files or stream.
shuffle has no effect after splitting	take and skip prevent later shuffle calls. Shuffle first.
A private dataset will not load	Pass token=True: load_dataset("org/name", token=True), after logging in.
No network	Set HF_HUB_OFFLINE=1; datasets already cached reload offline.
A text file did not upload	Files over 10MB are not committed; compress them.
Install or import errors	Check Python is 3.10+ and use a fresh virtual environment.

Alternatives to loading from the Hub

The same library works without the Hub, using the routes in the table above: local CSV, JSON, Parquet and other files, or data already in memory via from_pandas and from_dict. Data from the Hub and local data end up as the same Dataset type.

FAQ

How to use a Hugging Face dataset?

Install with pip install datasets, then call load_dataset with the dataset name, configuration and split, and pin revision= for reproducibility.

What is Hugging Face vs Kaggle?

See the section above: the documentation describes the Hugging Face side only, a Hub of versioned community datasets and a library to load them.

What is huggingface.co used for?

For datasets, it is the Hub where community datasets are hosted as git-based repositories with revisions and dataset cards, covering audio, vision and text.

How do I create datasets for machine learning?

Upload files through the website or call push_to_hub from Python, and add a dataset card with license, language and task categories.

Sources

Hugging Face Datasets documentation, v4.8.4, all read 2026-11-07: Installation, Quickstart, Load a dataset, Stream, Share a dataset to the Hub. Hub: huggingface.co/datasets.