dclm-baseline-1B

## Sampling Methodology

This dataset was created using **reservoir sampling**, a statistically unbiased random sampling algorithm that guarantees each sample from the source dataset has an equal probability of being included. This ensures the 1B token sample is representative of the full dataset's characteristics.

**Source Dataset**: [mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)
**Sample Size**: 1B tokens
**Content**: Filtered, diverse web content

Reservoir sampling enables rapid experimentation and ablation studies without processing the entire source dataset, while maintaining statistical validity of results.

For details on how this dataset was used in optimal pre-training data composition research, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

## Citation

If you use this model/dataset, please cite:

```bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}
```

For more details, see the [blog post](https://huggingface.co/blog/codelion/optimal-dataset-mixing/).

# 采样策略
本数据集采用**蓄水池采样（reservoir sampling）**构建而成，这是一种统计无偏随机采样算法，可确保源数据集中的每个样本被选中的概率均等。此举可保证该10亿Token（Token）样本能够完整反映源数据集的整体特征。

## 源数据集：[mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)
## 采样规模：10亿Token（Token）
## 内容：经过筛选的多样化网络文本内容

蓄水池采样可在无需处理完整源数据集的前提下，支持快速实验与消融研究，同时保证实验结果的统计有效性。

若需了解该数据集在最优预训练数据组合研究中的具体应用细节，请参阅[博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

## 引用信息
若您使用本模型/数据集，请引用如下文献：
bibtex
@article{sharma2025billion,
title={The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix},
author={Sharma, Asankhaya},
year={2025},
url={https://huggingface.co/blog/codelion/optimal-dataset-mixing/}
}

更多相关细节，请参阅[该博客文章](https://huggingface.co/blog/codelion/optimal-dataset-mixing/)。

搜集汇总

背景与挑战

背景概述

该数据集通过储层采样方法从mlfoundations/dclm-baseline-1.0源数据集中抽取了10亿个token的样本，内容为经过过滤的多样化网络数据。它旨在支持快速实验和消融研究，同时保持结果的统计有效性。