The problem
The useful sizes were missing
While running many different AI and machine learning experiments, I repeatedly found that the available dataset sizes did not match the work I wanted to do. The full dataset was often much larger than a research experiment needed, while the available small samples were too small or otherwise inconvenient.
What was missing was a middle ground: enough data for a meaningful experiment, but small enough to download, inspect, debug, and iterate on without committing the time and compute required by the full dataset.
The method
Smaller without losing the original mix
I downloaded the full source datasets I wanted to work with and created smaller subsets at sizes that were more useful for research. The goal was not simply to take the first records or produce an arbitrary sample.
Each subset was created to maintain the important data type, subject, and source ratios of its parent dataset. That keeps the smaller version representative of the larger collection instead of making convenience come at the cost of a distorted data mixture.
The collection
Research-sized versions of major datasets
The collection currently includes subsets based on FineWeb-Edu, Falcon RefinedWeb, Cosmopedia, and RedPajama. Available sizes include 100M, 1B, and 10B token variants, depending on the source dataset and the gap the subset was designed to fill.
- FineWeb-Edu with its CommonCrawl source distribution preserved.
- Cosmopedia sampled across its original content subsets.
- RedPajama variants that retain their source and domain mixture.
- Falcon RefinedWeb variants sized for smaller experiments.
Each Hugging Face dataset card documents its source, target size, creation process, and intended use.
The outcome
Faster experiments with representative data
Bite Size Data makes it easier to prototype models, debug training code, run CI workloads, and explore scaling behavior without beginning every experiment with the largest available dataset. The subsets are smaller on purpose, but their relationship to the source data is still treated as part of the dataset's design.