Common Crawl: Open Web Data Corpus Guide

Common Crawl is an open repository of web crawl data that has been routinely collected and published since 2008. Maintained as a non-profit initiative, it provides researchers with free public access to raw web archives, extracted plain text, and structural metadata. In computational linguistics, artificial intelligence, and web science, Common Crawl serves as an empirical foundation for large language models (LLMs) and web-scale text mining, eliminating the extensive infrastructure required to harvest web data independently. Managing these archives in compliance with FAIR data principles ensures that extraction pipelines remain findable, accessible, interoperable, and reusable.

What It Is

Common Crawl is a bulk web archive structured specifically for computational research, corpus linguistics, and machine learning. Rather than providing an interactive search engine or graphical interface, Common Crawl delivers periodic snapshot archives representing billions of web pages across hundreds of languages. The corpus captures static representations of public internet content, making longitudinal research data available to scholars worldwide.

To support varying analytical demands, Common Crawl packages its crawl archives into standardized file formats:

WARC (Web ARChive) Files: The raw capture format established under the ISO 28500 standard. WARC files contain complete transaction records, including raw HTTP request and response headers, MIME types, and original HTML content. In Summer 2013, Common Crawl transitioned from the legacy ARC format to WARC.
WET (Web Extraction Text) Files: Plain-text extracts derived directly from WARC files. HTML tags, navigation elements, scripts, and styling directives are stripped out, leaving the primary textual content. WET files serve as the standard starting artifact for language model pre-training.
WAT (Web Archive Transformation) Files: Structural metadata extracts stored as JSON objects. WAT files summarize HTTP headers, HTML tag hierarchies, document properties, and outgoing hyperlinks without retaining full page text, enabling network and graph analysis.
CDX / CDXJ Index Files: Lightweight columnar and text indexes mapping each crawled URL to its specific WARC filename, compressed byte offset, and record length.

Who Runs It and What It Covers

The corpus is maintained by the Common Crawl Foundation, a 501(c)(3) non-profit organization established in 2008 in Beverly Hills, California. Its charter focuses on democratizing web data access so that academic researchers and independent teams can analyze internet-scale corpora without commercial-scale crawling infrastructure.

Automated collection is conducted by CCBot, a distributed crawler built on Apache Nutch and Apache Hadoop Map-Reduce. CCBot discovers and retrieves public web pages under documented operational parameters:

User-Agent: Identified by the string CCBot/2.0 (https://commoncrawl.org/faq/) (with historical archives citing CCBot/1.0).
Robots Protocol: Complies with RFC 9309, checking domain robots.txt files and honoring Disallow and Crawl-delay instructions.
Protocols: Executes HTTP GET over HTTP/1.1 and HTTP/2 over TLS (HTTPS), supporting IPv4 and IPv6.
Redirects: Follows up to four consecutive HTTP redirects for web pages, and up to five when fetching robots.txt.
Polite Crawling: Employs an adaptive back-off algorithm when encountering HTTP 429 (Too Many Requests) or 5xx server responses, pausing several seconds between requests to the same site by default.
Browser Execution: CCBot does not execute client-side JavaScript and does not manage cookies. It stores initial server-side HTTP responses, meaning Single-Page Applications (SPAs) are captured in unrendered state.

The corpus is hosted as an Amazon Web Services (AWS) Public Dataset in the Amazon S3 bucket s3://commoncrawl/ located in the us-east-1 (Northern Virginia) region. Additional mirrors exist on academic cloud networks and Hugging Face.

How to Search or Use It

Because complete monthly crawls comprise hundreds of terabytes, researchers typically query indexes or execute targeted downloads. The documented retrieval protocol consists of seven steps:

Select a Crawl Archive: Choose a monthly snapshot identifier formatted as CC-MAIN-YYYY-WW from the Common Crawl release index.
Query the CDX Server API: Query specific domains or URL patterns using the CDX API endpoint at https://index.commoncrawl.org/CC-MAIN-YYYY-WW-index?url=example.com/*&output=json.
Extract Offset and Length Pointers: Parse the resulting JSON to obtain filename, offset (byte start position), and length (compressed byte count).
Fetch Target Records via HTTP Range Requests: Download specific records without full archive downloads using an HTTP byte-range header:
```
curl -H "Range: bytes=[offset]-[offset+length-1]" https://data.commoncrawl.org/[filename] -o target.warc.gz
```
Process at Scale in AWS us-east-1: For bulk processing, deploy EC2 or EMR compute in us-east-1 to read directly from s3://commoncrawl/ without egress fees:
```
aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-YYYY-WW/ --no-sign-request
```
Parse Payload Records: Read compressed files using WARC parsers, extracting headers and text according to established metadata standards.
Clean and Filter Corpus Text: Filter extracted WET text by applying language identification, removing boilerplate lines, and executing MinHash deduplication to eliminate repeated content.

Strengths and Limits

Applying Common Crawl to scientific research requires balancing substantial coverage against methodological boundaries:

Strengths:
- Broad Coverage: Longitudinal archives collected across more than fifteen years, covering billions of pages per crawl snapshot.
- Open Accessibility: Free public hosting via AWS Open Data with no subscription or licensing charges.
- Multi-Format Partitioning: Independent WARC, WET, and WAT distributions allow targeted ingestion matching specific research needs.
- Integration: Native compatibility with cloud-scale data frameworks including Spark, Ray, and Hugging Face.
Limits:
- JavaScript Omission: Dynamic text rendered exclusively via client-side JavaScript is not captured by CCBot.
- Corpus Noise: Raw crawl data contains duplicate syndications, boilerplate templates, and low-quality web text requiring rigorous post-filtering.
- Sampling Nature: Captures a broad sample of the open web rather than complete site mirrors.
- Compute Costs: While data access is free, cloud computing in AWS and cross-region network transfers carry vendor infrastructure costs.
- Copyright Governance: Common Crawl distributes data under its Terms of Use (updated March 7, 2024), governed by California law. However, third-party content within archives remains subject to original copyright; researchers rely on fair use or text-and-data-mining exemptions.

For comparative extraction strategies, research teams can contrast bulk corpus ingestion with targeted web scraping methodologies.

Alternatives

Different computational goals benefit from distinct corpus architectures. The table below outlines key alternatives to Common Crawl:

Corpus / Platform	Primary Scope	Access Route	Data Granularity	Typical Research Application
Common Crawl	Broad monthly snapshots of open web collected since 2008.	AWS S3 (`s3://commoncrawl/`), HTTP Range requests, CDX API.	Raw WARC, plain-text WET, JSON metadata WAT.	Pre-training large language models, web structure analysis, and linguistic corpora.
Internet Archive Wayback Machine	Historical web archiving focused on page preservation and temporal revisions.	Wayback Machine API, CDX server, and web portal.	Historical page captures and interactive WARC replays.	Digital humanities, historical URL verification, and longitudinal site evolution.
Wikipedia Dumps	Curated encyclopedic articles with structured infoboxes.	XML/SQL database dump downloads via Wikimedia.	Structured text, revision histories, and category trees.	Knowledge graph construction, factual question answering, and entity recognition.
C4 (Colossal Clean Crawled Corpus)	Filtered and cleaned English web text derived from Common Crawl’s April 2019 crawl.	Hugging Face Datasets and TensorFlow Datasets.	Pre-cleaned, heuristic-filtered plain text.	NLP model benchmarking and standardized baseline training (e.g., T5).
FineWeb / Dolma	Open-source pre-training datasets built from filtered Common Crawl snapshots.	Hugging Face Hub and open cloud storage.	Deduplicated, quality-annotated tokenized text.	Auditable language model pre-training with open filtering recipes.

FAQ

What is the difference between WARC, WET, and WAT files?
WARC files store complete raw crawl records including HTTP request and response headers and full HTML payloads. WET files store extracted plain text stripped of HTML tags, navigation, and formatting. WAT files store structural metadata in JSON format, capturing HTTP headers, page metrics, and outbound links without preserving page text.

Does CCBot execute client-side JavaScript or process cookies?
No. CCBot is an automated HTTP crawler that performs standard HTTP GET requests. It does not run a browser engine to execute JavaScript or maintain session cookies, capturing only the initial server-delivered response.

How can researchers query specific URLs without downloading multi-terabyte files?
Researchers query the CDX Index Server API to find a URL’s archive filename, byte offset, and compressed record length. They can then issue an HTTP Range request using command-line tools like curl to download only the target byte slice.

Under what terms is Common Crawl data released?
Common Crawl data is governed by the platform’s Terms of Use (last updated March 7, 2024), governed by California law. The platform applies minimal restrictions to facilitate research, but it does not license the underlying third-party web content, which remains subject to copyright and fair use principles.

How can website administrators opt out of Common Crawl?
Site administrators can add disallow rules for CCBot in their domain’s robots.txt file in accordance with RFC 9309. Additionally, domain owners can register on the Common Crawl opt-out registry to request exclusion from subsequent crawl runs.

Sources

Common Crawl. “Overview.” https://commoncrawl.org/overview (checked November 7, 2026).
Common Crawl. “Frequently Asked Questions.” https://commoncrawl.org/faq (checked November 7, 2026).
Common Crawl. “Get Started.” https://commoncrawl.org/get-started (checked November 7, 2026).
Common Crawl. “Terms of Use.” https://commoncrawl.org/terms-of-use (last updated March 7, 2024; checked November 7, 2026).