rajat5039/wiki-multihop-qa-500k

--- language: - en license: cc-by-4.0 task_categories: - question-answering task_ids: - extractive-qa - open-domain-qa tags: - multi-hop - reasoning - synthetic - wikipedia - chain-of-thought - latent-reasoning - think-in-silence size_categories: - 100K<n<1M --- # wiki-multihop-qa-500k **500,000 synthetic multi-hop QA pairs generated from Wikipedia.** Built for training the [think-in-silence](https://github.com/rajatmalik/think-in-silence) latent reasoning model — a model that reasons entirely in vector space without generating chain-of-thought tokens. --- ## Why This Dataset Exists Most publicly available QA datasets have two problems for reasoning research: 1. **Too small.** HotpotQA has 113K samples. StrategyQA has 2.8K. Not enough diversity to train a generalizable reasoning module. 2. **Too many single-hop questions.** A question answerable from one sentence teaches the model that one thinking step is enough. That directly breaks the K-scaling property we're trying to demonstrate. This dataset was built to fix both. Every pair requires connecting at least two facts. The model must reason, not look up. --- ## What Makes a Good Multi-Hop Question **Single-hop (NOT in this dataset):** ``` Q: Where was Marie Curie born? A: Warsaw ``` One fact. One lookup. No reasoning chain needed. **Multi-hop (IN this dataset):** ``` Q: In which city did the scientist born in Warsaw later discover polonium? A: Paris ``` Two facts connected: (1) Curie was born in Warsaw → (2) she worked in Paris → (3) discovered polonium there. This is the reasoning pattern that requires K > 1 thinking steps in the think-in-silence architecture. --- ## Dataset Statistics | Split | Samples | |------------|-----------| | Train | ~450,000 | | Validation | ~25,000 | | Test | ~25,000 | | **Total** | **~500,000** | ### Difficulty Distribution (Train) | Score | Type | Description | Approx % | |-------|-------------|------------------------------------|----------| | 0 | single-hop | One fact lookup | ~5% | | 1 | two-hop | Connect two facts | ~55% | | 2 | multi-hop | Three or more connections required | ~40% | > Multi-hop ratio (score ≥ 1): **~95% of the dataset** ### Answer Type Distribution | Type | Description | Example | |---------|--------------------------------|----------------| | entity | Named person, place, or thing | "Paris" | | phrase | Short descriptive phrase | "Atlantic Ocean"| | numeric | Number or quantity | "1867" | | date | Year or date | "1905" | | boolean | Yes/No answer | "yes" | --- ## Data Fields Each sample contains: ```python { "question": str, # Multi-hop question requiring 2+ facts "answer": str, # Short answer (1-6 words) "hops": int, # Number of reasoning hops (always >= 2) "difficulty_score": int, # 0=single-hop, 1=two-hop, 2=multi-hop "answer_type": str, # entity | phrase | numeric | date | boolean "source": str, # Wikipedia chunk file the paragraph came from } ``` ### Example Samples ```json { "question": "In which city did the physicist born in Warsaw conduct her Nobel Prize-winning research?", "answer": "Paris", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0042.jsonl" } { "question": "What element was discovered by the scientist who also founded the Radium Institute in France?", "answer": "Polonium", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0042.jsonl" } { "question": "Which country's capital city was home to the university where the Nobel Prize winner who was born in 1867 studied?", "answer": "France", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0089.jsonl" } ``` --- ## Generation Pipeline ### Source Corpus - **500,000 English Wikipedia articles** streamed from `wikimedia/wikipedia` (20231101.en) - Split into **3,308,934 paragraphs** (30–200 words each) - Stored as 331 JSONL chunk files ### Generation Model - **Gemini 2.5 Flash-Lite** via Google AI API - 10 concurrent workers - ~21 pairs/second sustained throughput ### Prompt Design The generation prompt was carefully engineered to force multi-hop output: ``` STRICT RULES: 1. Each question MUST require connecting 2+ facts from the passage 2. Single-fact questions ("Where was X born?") are REJECTED 3. Answers: 1-6 words maximum 4. Output ONLY a JSON array GOOD: connects 2 facts BAD: only 1 fact needed — REJECTED ``` The good/bad example in the prompt was critical — without it, the model generated mostly single-hop questions despite the instructions. ### Quality Filtering Every generated pair passed through 8 filters before being kept: | Filter | What it removes | |--------|----------------| | `single_hop` | hops < 2 | | `question_too_short` | fewer than 6 words | | `no_question_mark` | missing ? | | `bad_question_start` | doesn't start with question word | | `answer_too_vague` | "yes", "no", "it", "they" | | `answer_too_long` | more than 15 words | | `answer_in_question` | trivial lookup | | `question_has_artifact` | generation artifacts | Plus near-deduplication (Jaccard similarity > 0.85 threshold). ### Generation Stats ``` Raw pairs generated: 1,050,000 After quality filter: ~925,000 (88%) After difficulty filter: ~504,000 (48% overall retention) After deduplication: ~500,000 Total API cost: ~$35 Generation time: ~18 hours ``` --- ## How to Use ### Basic Loading ```python from datasets import load_dataset dataset = load_dataset("rajatmalik/wiki-multihop-qa-500k") # Access splits train = dataset["train"] val = dataset["validation"] test = dataset["test"] print(train[0]) # { # "question": "In which city did the physicist born in Warsaw...", # "answer": "Paris", # "hops": 2, # "difficulty_score": 2, # "answer_type": "entity", # "source": "chunk_0042.jsonl" # } ``` ### Filter by Difficulty ```python # Only hardest multi-hop (score=2) hard = dataset["train"].filter(lambda x: x["difficulty_score"] == 2) # Only two-hop (score=1) medium = dataset["train"].filter(lambda x: x["difficulty_score"] == 1) ``` ### Filter by Answer Type ```python # Only entity answers entities = dataset["train"].filter(lambda x: x["answer_type"] == "entity") # Only numeric numeric = dataset["train"].filter(lambda x: x["answer_type"] == "numeric") ``` ### Combine with Public Datasets ```python from datasets import load_dataset, concatenate_datasets wiki_mhop = load_dataset("rajatmalik/wiki-multihop-qa-500k", split="train") hotpotqa = load_dataset("hotpot_qa", "distractor", split="train") # Use together for training combined = concatenate_datasets([wiki_mhop, hotpotqa]) ``` --- ## Intended Use ### Primary Use — think-in-silence Training This dataset was built specifically to train the ThoughtModule in [think-in-silence](https://github.com/rajatmalik/think-in-silence) — a latent reasoning model that performs K recurrent cross-attention steps in a 256-dimensional space. Multi-hop questions are essential because: - Single-hop questions teach the model K=1 is enough - Multi-hop questions force K > 1 thinking steps - The K-scaling property only emerges with sufficient multi-hop training signal ### Other Suitable Uses - Training retrieval-augmented generation (RAG) systems - Fine-tuning LLMs for multi-step reasoning - Evaluating question answering systems - Research on chain-of-thought and reasoning ### Not Suitable For - Factual question answering benchmarks (answers are synthetic, may contain errors) - Tasks requiring long-form answers - Non-English tasks --- ## Limitations **Answer accuracy is not guaranteed.** This is a synthetically generated dataset. Gemini 2.5 Flash-Lite may occasionally generate incorrect answers or misattribute facts. For research purposes, the reasoning structure (multi-hop) matters more than factual accuracy. **Wikipedia coverage.** All questions are grounded in Wikipedia (November 2023 snapshot). Topics not well-covered in Wikipedia are underrepresented. **English only.** The source corpus is English Wikipedia. All questions and answers are in English. **Retention rate.** ~48% of generated pairs passed all filters. The majority of rejections were single-hop questions that slipped through despite the prompt instructions. This means the remaining dataset is high-confidence multi-hop. --- ## Comparison to Related Datasets | Dataset | Size | Multi-hop | Synthetic | Free | |----------------|--------|-----------|-----------|------| | HotpotQA | 113K | ✓ | ✗ | ✓ | | StrategyQA | 2.8K | ✓ | ✗ | ✓ | | MuSiQue | 20K | ✓ | ✗ | ✓ | | **wiki-multihop-qa-500k** | **500K** | **✓** | **✓** | **✓** | The key advantage is scale — 500K multi-hop pairs versus the largest public alternative at 113K. --- ## Generation Code Full pipeline code is open source: ``` https://github.com/rajatmalik/think-in-silence-data ``` To reproduce this dataset: ```bash git clone https://github.com/rajatmalik/think-in-silence-data cd think-in-silence-data pip install -r requirements.txt export GOOGLE_API_KEY=your_key python run.py --yes ``` --- ## Citation If you use this dataset in your research, please cite: ```bibtex @dataset{malik2026wikimultihop, author = {Malik, Rajat}, title = {wiki-multihop-qa-500k: Synthetic Multi-Hop QA from Wikipedia}, year = {2026}, publisher = {HuggingFace}, url = {https://huggingface.co/datasets/rajatmalik/wiki-multihop-qa-500k}, note = {Generated using Gemini 2.5 Flash-Lite from 500K Wikipedia articles} } ``` --- ## Related Work - **think-in-silence** — The model this dataset was built for: [github.com/rajatmalik/think-in-silence](https://github.com/rajatmalik/think-in-silence) - **I-JEPA** (Assran et al., 2023) — JEPA training objective this project extends to language - **Coconut** (Hao et al., 2024) — Related latent reasoning approach - **HotpotQA** (Yang et al., 2018) — Original multi-hop QA dataset --- ## License [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) You are free to use, share, and adapt this dataset for any purpose, including commercial use, as long as you give appropriate credit. --- *Built by Rajat Malik · 2026 · Part of the think-in-silence research project*--- 语言： - 英语 许可协议：cc-by-4.0 任务类别： - 问答（question-answering） 任务子类型： - 抽取式问答（extractive-qa） - 开放域问答（open-domain-qa） 标签： - 多跳推理（multi-hop） - 推理（reasoning） - 合成数据集（synthetic） - 维基百科（wikipedia） - 思维链（chain-of-thought） - 潜在推理（latent-reasoning） - 静默思考（think-in-silence） 样本规模： - 10万<样本数<100万 --- # wiki-multihop-qa-500k **50万条源自维基百科的合成多跳问答样本对** 本数据集专为训练[静默思考（think-in-silence）](https://github.com/rajatmalik/think-in-silence)潜在推理模型而构建——该模型完全在向量空间中完成推理，无需生成思维链（chain-of-thought）Token。 --- ## 数据集构建初衷 当前公开可用的问答数据集在推理研究中存在两大痛点： 1. **样本规模不足**：HotpotQA仅含11.3万条样本，StrategyQA仅2.8千条，多样性不足以训练通用化的推理模块。 2. **单跳问题占比过高**：仅需单句即可作答的问题会让模型认为仅需一步思考即可完成任务，这直接破坏了我们拟验证的K步缩放特性。 本数据集旨在解决上述两大问题：所有样本均要求关联至少两条事实，模型必须通过推理才能得到答案，而非仅通过检索。 --- ## 合格多跳问答的判定标准 **单跳问题（不在本数据集内）：** Q: 玛丽·居里出生于何地？ A: 华沙 仅需单条事实、单次检索，无需推理链。 **多跳问题（纳入本数据集）：** Q: 出生于华沙的科学家后来在哪座城市发现了钋元素？ A: 巴黎 该问题关联了三条事实：(1) 居里夫人出生于华沙；(2) 她曾在巴黎工作；(3) 她在此处发现了钋元素。这一推理模式要求在静默思考（think-in-silence）架构中执行K>1步的思考步骤。 --- ## 数据集统计信息 | 数据集划分 | 样本数量 | |------------|----------| | 训练集 | 约45万 | | 验证集 | 约2.5万 | | 测试集 | 约2.5万 | | **总计** | **约50万**| ### 训练集难度分布 | 难度得分 | 类型 | 说明 | 占比近似值 | |----------|------------|---------------------------------------|------------| | 0 | 单跳问题 | 仅需单条事实检索 | ~5% | | 1 | 两跳问题 | 需关联两条事实 | ~55% | | 2 | 多跳问题 | 需关联三条及以上事实 | ~40% | > 多跳问题占比（得分≥1）：**数据集的95%左右** ### 答案类型分布 | 答案类型 | 说明 | 示例 | |---------|-----------------------|---------------------| | 实体 | 命名的人物、地点或事物 | "巴黎" | | 短语 | 简短描述性短语 | "大西洋" | | 数值 | 数字或数量 | "1867" | | 日期 | 年份或具体日期 | "1905" | | 布尔值 | 是/否类答案 | "是"、"否" | --- ## 数据字段说明 每个样本包含以下字段： python { "question": str, # 多跳问题，需关联2条及以上事实 "answer": str, # 简短答案（1-6个词） "hops": int, # 推理跳数（始终≥2） "difficulty_score": int, # 难度得分：0=单跳，1=两跳，2=多跳 "answer_type": str, # 答案类型：entity | phrase | numeric | date | boolean "source": str, # 该段落所属的维基百科分块文件 } ### 样本示例 json { "question": "出生于华沙的物理学家在哪座城市开展了其诺贝尔获奖研究？", "answer": "巴黎", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0042.jsonl" } { "question": "在法国创立镭研究所的科学家发现了哪种元素？", "answer": "钋", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0042.jsonl" } { "question": "1867年出生的诺贝尔奖得主曾就读的大学所在国家的首都是哪里？", "answer": "法国", "hops": 2, "difficulty_score": 2, "answer_type": "entity", "source": "chunk_0089.jsonl" } --- ## 生成流程 ### 源语料库 - **50万篇英文维基百科文章**：源自`wikimedia/wikipedia`（20231101.en）快照 - 被切分为**3,308,934个段落**（每个段落30-200词） - 存储为331个JSONL分块文件 ### 生成模型 - 通过Google AI API调用**Gemini 2.5 Flash-Lite** - 采用10个并发工作进程，持续吞吐量约为21对/秒 ### 提示词设计 生成提示词经过精心设计，强制生成多跳问答样本： STRICT RULES: 1. 每个问题必须关联段落中的2条及以上事实 2. 单事实问题（如“X出生于何地？”）将被拒绝 3. 答案长度不超过1-6个词 4. 仅输出JSON数组 GOOD: 关联2条事实 BAD: 仅需1条事实 —— 拒绝生成 提示词中的好坏示例至关重要——若无该示例，即使添加指令，模型仍会生成大量单跳问题。 ### 质量过滤 所有生成的样本对在最终保留前需经过8轮过滤： | 过滤规则 | 移除内容 | |--------|----------------| | `single_hop` | 推理跳数<2的样本 | | `question_too_short` | 问题词数少于6的样本 | | `no_question_mark` | 缺失问号的样本 | | `bad_question_start` | 不以疑问词开头的样本 | | `answer_too_vague` | 答案为“yes”、“no”、“it”、“they”等模糊表述的样本 | | `answer_too_long` | 答案词数超过15的样本 | | `answer_in_question` | 答案可直接从问题中获取的样本 | | `question_has_artifact` | 包含生成 artifacts 的样本 | 此外还加入了近重复过滤（Jaccard相似度阈值>0.85）。 ### 生成统计 原始生成样本对： 1,050,000 质量过滤后： ~925,000 （保留率88%） 难度过滤后： ~504,000 （整体保留率48%） 去重后： ~500,000 总API调用成本： ~$35 生成总耗时： ~18小时 --- ## 使用方法 ### 基础加载 python from datasets import load_dataset dataset = load_dataset("rajatmalik/wiki-multihop-qa-500k") # 访问数据集划分 train = dataset["train"] val = dataset["validation"] test = dataset["test"] print(train[0]) # { # "question": "出生于华沙的物理学家在哪座城市开展了其诺贝尔获奖研究？", # "answer": "巴黎", # "hops": 2, # "difficulty_score": 2, # "answer_type": "entity", # "source": "chunk_0042.jsonl" # } ### 按难度过滤 python # 仅保留难度最高的多跳样本（得分=2） hard = dataset["train"].filter(lambda x: x["difficulty_score"] == 2) # 仅保留两跳样本（得分=1） medium = dataset["train"].filter(lambda x: x["difficulty_score"] == 1) ### 按答案类型过滤 python # 仅保留实体类答案 entities = dataset["train"].filter(lambda x: x["answer_type"] == "entity") # 仅保留数值类答案 numeric = dataset["train"].filter(lambda x: x["answer_type"] == "numeric") ### 与公开数据集结合使用 python from datasets import load_dataset, concatenate_datasets wiki_mhop = load_dataset("rajatmalik/wiki-multihop-qa-500k", split="train") hotpotqa = load_dataset("hotpot_qa", "distractor", split="train") # 合并用于训练 combined = concatenate_datasets([wiki_mhop, hotpotqa]) --- ## 预期用途 ### 主要用途——静默思考模型训练 本数据集专为训练[think-in-silence](https://github.com/rajatmalik/think-in-silence)中的思考模块而构建，该模型在256维空间中执行K轮循环交叉注意力步骤的潜在推理模型。多跳问题至关重要，因为： - 单跳问题会让模型认为仅需K=1步思考 - 多跳问题强制模型执行K>1步的思考步骤 - K步缩放特性仅在充足的多跳训练信号下才能显现 ### 其他适用场景 - 训练检索增强生成（retrieval-augmented generation, RAG）系统 - 微调大语言模型（LLM/Large Language Model）以提升多步推理能力 - 评估问答系统性能 - 开展思维链与推理相关研究 ### 不适用场景 - 事实问答基准测试（答案为合成生成，可能存在错误） - 需要长格式答案的任务 - 非英语语言任务 --- ## 数据集局限性 **答案准确性无法保证**：本数据集为合成生成，Gemini 2.5 Flash-Lite可能偶尔生成错误答案或错误关联事实。对于研究而言，推理结构（多跳特性）比事实准确性更为重要。 **维基百科覆盖范围**：所有问题均基于2023年11月快照的维基百科内容，维基百科覆盖不足的主题在数据集中占比偏低。 **仅支持英语**：源语料库为英文维基百科，所有问题与答案均为英语。 **保留率**：仅约48%的生成样本对通过全部过滤规则。大部分被拒样本为尽管有提示词限制但仍生成的单跳问题，这意味着剩余数据集为高置信度的多跳问答样本。 --- ## 与相关数据集的对比 | 数据集名称 | 样本规模 | 支持多跳 | 是否合成 | 免费 | |----------------|--------|-----------|-----------|------| | HotpotQA | 11.3万 | ✓ | ✗ | ✓ | | StrategyQA | 2.8千 | ✓ | ✗ | ✓ | | MuSiQue | 2万 | ✓ | ✗ | ✓ | | **wiki-multihop-qa-500k** | **50万** | **✓** | **✓** | **✓** | 本数据集的核心优势为规模优势——拥有50万条多跳问答样本，远大于当前最大的公开同类数据集（11.3万条）。 --- ## 生成代码 完整的生成流程代码已开源： https://github.com/rajatmalik/think-in-silence-data 复现该数据集的步骤如下： bash git clone https://github.com/rajatmalik/think-in-silence-data cd think-in-silence-data pip install -r requirements.txt export GOOGLE_API_KEY=your_key python run.py --yes --- ## 引用格式 如果在研究中使用本数据集，请引用如下内容： bibtex @dataset{malik2026wikimultihop, author = {Malik, Rajat}, title = {wiki-multihop-qa-500k: Synthetic Multi-Hop QA from Wikipedia}, year = {2026}, publisher = {HuggingFace}, url = {https://huggingface.co/datasets/rajatmalik/wiki-multihop-qa-500k}, note = {使用Gemini 2.5 Flash-Lite基于50万篇维基百科文章生成} } --- ## 相关研究 - **静默思考（think-in-silence）**：本数据集专为训练的模型：[github.com/rajatmalik/think-in-silence](https://github.com/rajatmalik/think-in-silence) - **I-JEPA**（Assran等人，2023）：本项目扩展至语言领域的JEPA训练目标 - **Coconut**（Hao等人，2024）：相关的潜在推理研究方法 - **HotpotQA**（Yang等人，2018）：首个多跳问答公开数据集 --- ## 许可协议 [知识共享署名4.0（CC BY 4.0）](https://creativecommons.org/licenses/by/4.0/) 您可自由使用、分享和改编本数据集，包括商业用途，只需给予适当的署名即可。 --- *由Rajat Malik构建 · 2026年 · 属于think-in-silence研究项目的一部分*