The Pile

pile.eleuther.ai AI Data & Analytics Dataset

The Pile dataset combines 22 smaller datasets for language modeling research.

About The Pile

The Pile is a text dataset for training and evaluating language models. It is 825 GiB in size and combines 22 smaller datasets. The homepage describes the sources as diverse and high quality, and lists books, GitHub repositories, webpages, chat logs, and papers in medical, physics, math, computer science, and philosophy.

The Pile is downloaded from a host called The Eye, in jsonlines format compressed with zstandard. The homepage also describes Pile BPB, a benchmark for cross-domain text modeling, and shows a leaderboard of model scores. The leaderboard notes possible test-set overlap for some entries and states that some zero-shot results were obtained without all components of the Pile in the training data. Pricing is not known.

Key features

825 GiB text dataset combining 22 smaller datasets
Sources include books, GitHub repositories, webpages, and chat logs
Includes papers in medical, physics, math, computer science, and philosophy
Distributed as zstandard-compressed jsonlines files
Pile BPB benchmark for cross-domain text modeling
Leaderboard of model Test BPB scores with evaluation code

Best for: Researchers and teams who train or evaluate large language models and need a broad, multi-domain text corpus.

Good for

Combines many different text domains, which the homepage says can improve cross-domain knowledge in large models
Provides a published benchmark, Pile BPB, for comparing models on general text modeling

Watch out for

The 825 GiB size makes the full download large
Leaderboard entries may include test-set overlap, and some zero-shot results were trained without all components of the Pile