krisbailey/RedPajama-10B-Weighted

---
license: apache-2.0
task_categories:
- text-generation
language:
- en
tags:
- redpajama
- llm
- dataset-reproduction
- redpajama-10b
- redpajama-subset
- redpajama-weighted
- redpajama-sample
- natural-language-processing
size_categories:
- 1B<n<10B
pretty_name: RedPajama 10B Weighted Subset
---

# RedPajama-10B-Weighted

A **canonical 10 Billion token weighted subset** of the [RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) dataset.

## Dataset Description

This dataset is a faithful reproduction of the original RedPajama-Data-1T distribution, scaled down to exactly **10 Billion tokens**. It is designed to preserve the **exact domain ratios** of the original dataset (excluding the defunct 'Books' subset). This allows researchers and developers to prototype, debug, and test on a representative slice of the data without needing to download or process the full 1 Terabyte dataset.

It serves as both a standalone dataset for medium-scale experiments and the parent source for smaller slices (like the [1B subset](https://huggingface.co/datasets/krisbailey/RedPajama-1B-Weighted)).

## Dataset Details
- **Total Tokens:** ~10,000,000,000 (10 Billion)
- **Source:** [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T)
- **Language:** English
- **Format:** Apache Parquet
- **Producer:** Kris Bailey

## Motivation
The original RedPajama dataset is a standard for open-source LLM training, but its size (1TB+) makes it unwieldy for quick iteration, debugging, or educational purposes. Randomly sampling without care can destroy the delicate balance of data sources (CommonCrawl vs. C4 vs. GitHub).

This **RedPajama 10B subset** solves this by using a **weighted interleaving strategy** that strictly adheres to the original mixing ratios. It ensures that even at a smaller scale, the data seen by the model is distributionally equivalent to the full run.

## Dataset Creation Process

The creation process involved a precise streaming and interleaving pipeline:

### 1. Source Streaming
We streamed the data directly from `togethercomputer/RedPajama-Data-1T` to avoid local storage bottlenecks.

### 2. Weighted Interleaving
We defined the target probabilities based on the original token counts:
- **CommonCrawl:** 74.16%
- **C4:** 14.78%
- **GitHub:** 4.98%
- **ArXiv:** 2.36%
- **Wikipedia:** 2.03%
- **StackExchange:** 1.69%

An interleaving algorithm sampled from these streams according to these probabilities to construct a single, unified stream.

### 3. Buffer Shuffling
To avoid burstiness (e.g., seeing 1000 Wikipedia articles in a row), we implemented a **buffer shuffle** with a size of 10,000 documents. This ensures a healthy mixture of domains throughout the dataset.

### 4. Verification
The process ran until exactly 10 Billion tokens were collected. We verified that the final composition matches the target weights.

## Composition

| Subset | Weight | Approx. Tokens |
| :--- | :--- | :--- |
| **CommonCrawl** | 74.16% | ~7.42 B |
| **C4** | 14.78% | ~1.48 B |
| **GitHub** | 4.98% | ~0.50 B |
| **ArXiv** | 2.36% | ~0.24 B |
| **Wikipedia** | 2.03% | ~0.20 B |
| **StackExchange** | 1.69% | ~0.17 B |

## Usage
```python
from datasets import load_dataset

# Load the 10B weighted subset
ds = load_dataset("krisbailey/RedPajama-10B-Weighted", split="train")

print(ds)
```

## Citation
If you use this dataset, please cite the original RedPajama work:
```bibtex
@software{together2023redpajama,
author = {Together Computer},
title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},
month = April,
year = 2023,
url = {https://github.com/togethercomputer/RedPajama-Data}
}
```

许可证：Apache-2.0
任务类别：文本生成
语言：英语
标签：
- redpajama
- 大语言模型（Large Language Model）
- 数据集复现
- redpajama-10b
- redpajama-subset
- redpajama-weighted
- redpajama-sample
- 自然语言处理
规模类别：10亿 < Token（Token）数 < 100亿
美观名称：RedPajama 10B Weighted Subset

# RedPajama-10B-Weighted

本数据集为[RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T)的标准100亿Token（Token）加权子集。

## 数据集说明
本数据集忠实复现了原始RedPajama-Data-1T数据集的分布，并被精简至恰好100亿Token。其设计目标是保留原始数据集（已失效的'Books'子集除外）的精确领域占比，使研究人员与开发者能够在无需下载或处理完整1TB数据集的前提下，基于具有代表性的数据切片进行原型开发、调试与测试。

本数据集既可作为面向中等规模实验的独立数据集，也可作为更小切片（如[1B子集](https://huggingface.co/datasets/krisbailey/RedPajama-1B-Weighted)）的父级数据源。

## 数据集详情
- **总Token数**：约100亿（10 Billion）
- **数据源**：[togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T)
- **语言**：英语
- **格式**：Apache Parquet
- **制作者**：Kris Bailey

## 设计动机
原始RedPajama数据集是开源大语言模型（Large Language Model）训练的标准基准，但因其规模（1TB以上），在快速迭代、调试或教学场景中使用较为不便。无规划的随机采样会破坏各数据源（CommonCrawl、C4与GitHub）间的微妙平衡。

本**RedPajama 10B子集**通过采用严格遵循原始混合比例的**加权交错策略**解决了上述问题，确保即使在较小规模下，模型所接触的数据分布也与完整数据集保持一致。

## 数据集构建流程
数据集构建采用了精准的流式传输与交错流水线：
### 1. 数据源流式传输
我们直接从`togethercomputer/RedPajama-Data-1T`流式读取数据，以规避本地存储瓶颈。
### 2. 加权交错
我们基于原始Token数量定义了目标概率：
- CommonCrawl：74.16%
- C4：14.78%
- GitHub：4.98%
- ArXiv：2.36%
- Wikipedia：2.03%
- StackExchange：1.69%
通过交错算法按照上述概率从各数据流中采样，构建出单一统一的数据流。
### 3. 缓冲区洗牌
为避免数据扎堆现象（例如连续出现1000篇维基百科文章），我们实现了大小为10000条文档的**缓冲区洗牌**机制，确保整个数据集中各领域的分布均衡。
### 4. 验证
整个流程持续至恰好收集到100亿Token为止，并验证最终数据组成与目标权重完全匹配。

## 数据集组成
| 子数据集 | 权重 | 近似Token数 |
| :--- | :--- | :--- |
| **CommonCrawl** | 74.16% | ~74.2亿 |
| **C4** | 14.78% | ~14.8亿 |
| **GitHub** | 4.98% | ~5.0亿 |
| **ArXiv** | 2.36% | ~2.4亿 |
| **Wikipedia** | 2.03% | ~2.0亿 |
| **StackExchange** | 1.69% | ~1.7亿 |

## 使用示例
python
from datasets import load_dataset

# 加载10B加权子集
ds = load_dataset("krisbailey/RedPajama-10B-Weighted", split="train")

print(ds)

## 引用
若使用本数据集，请引用原始RedPajama相关研究：
bibtex
@software{together2023redpajama,
author = {Together Computer},
title = {RedPajama：复现LLaMA训练数据集的开源方案},
month = 四月,
year = 2023,
url = {https://github.com/togethercomputer/RedPajama-Data}
}