juliannunezb/mixed-pretrain-100b-gpt2

Mixed-Pretrain-100B-GPT2 是一个包含1000亿个token的预训练混合数据集，使用GPT-2 BPE分词器（词汇表大小为50257）进行分词，并打包成flat np.memmap就绪的uint16二进制文件。该数据集设计用于参数规模在5亿到50亿之间的GPT-2词汇表模型的通用预训练语料库，旨在提供教育密度、事实广度、科学推理、数学、叙事多样性和合成高质量文本的平衡混合。数据集由多个来源组成，包括FineWeb-Edu、FineWeb、Cosmopedia v2、peS2o、Open-Web-Math、Wikipedia、FineMath和PG19书籍，总token数约为1000亿。数据集格式为每个来源槽位写入独立的flat uint16二进制文件，总磁盘占用约200GB。使用案例包括预训练500M到5B参数模型，并支持通过Chinchilla优化进行过训练。数据集基于Apache-2.0许可证发布，上游来源保留各自许可证。已知限制包括仅英语内容、仅支持GPT-2 BPE分词器、无代码数据、无多轮对话/指令数据，以及部分来源数据量不足目标。

Mixed-Pretrain-100B-GPT2 is a 100-billion-token pretrain mix tokenized with the GPT-2 BPE tokenizer (vocab=50257) and packed into flat np.memmap-ready uint16 binary files. Designed as a universal pretrain corpus for GPT-2-vocab models in the 500M to 5B parameter range, it is carefully balanced for educational density, factual breadth, scientific reasoning, math, narrative diversity, and synthetic high-quality text. The dataset is composed of multiple sources including FineWeb-Edu, FineWeb, Cosmopedia v2, peS2o, Open-Web-Math, Wikipedia, FineMath, and PG19 books, totaling approximately 100 billion tokens. The format consists of independent flat uint16 binary files per source slot, with a total disk footprint of ~200 GB. Use cases include pretraining 500M to 5B parameter models, supporting overtraining via Chinchilla optimal. The dataset is released under Apache-2.0 license, with upstream sources retaining their respective licenses. Known limitations include English-only content, GPT-2 BPE tokenizer only, no code data, no multi-turn chat/instruction data, and some source datasets falling short of target sizes.