hynky/okapi_arc_challenge

--- dataset_info: - config_name: ar features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 866140 num_examples: 773 download_size: 334900 dataset_size: 866140 - config_name: bn features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1392045 num_examples: 781 download_size: 427655 dataset_size: 1392045 - config_name: ca features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 637317 num_examples: 777 download_size: 285221 dataset_size: 637317 - config_name: da features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 597726 num_examples: 781 download_size: 269773 dataset_size: 597726 - config_name: de features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 673371 num_examples: 788 download_size: 297548 dataset_size: 673371 - config_name: es features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 660712 num_examples: 789 download_size: 292004 dataset_size: 660712 - config_name: eu features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 636227 num_examples: 774 download_size: 302334 dataset_size: 636227 - config_name: fr features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 701928 num_examples: 787 download_size: 302523 dataset_size: 701928 - config_name: gu features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1204342 num_examples: 731 download_size: 387499 dataset_size: 1204342 - config_name: hi features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1398051 num_examples: 773 download_size: 415177 dataset_size: 1398051 - config_name: hr features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 578523 num_examples: 769 download_size: 279158 dataset_size: 578523 - config_name: hu features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 653747 num_examples: 771 download_size: 303830 dataset_size: 653747 - config_name: hy features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 703742 num_examples: 553 download_size: 284970 dataset_size: 703742 - config_name: id features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 650389 num_examples: 778 download_size: 266654 dataset_size: 650389 - config_name: it features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 649613 num_examples: 783 download_size: 290933 dataset_size: 649613 - config_name: kn features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1232083 num_examples: 678 download_size: 389616 dataset_size: 1232083 - config_name: ml features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1477816 num_examples: 694 download_size: 450760 dataset_size: 1477816 - config_name: mr features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1410185 num_examples: 764 download_size: 438986 dataset_size: 1410185 - config_name: ne features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1396548 num_examples: 774 download_size: 430638 dataset_size: 1396548 - config_name: nl features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 643358 num_examples: 785 download_size: 284342 dataset_size: 643358 - config_name: pt features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 647883 num_examples: 788 download_size: 287376 dataset_size: 647883 - config_name: ro features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 655703 num_examples: 779 download_size: 292661 dataset_size: 655703 - config_name: ru features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1047861 num_examples: 788 download_size: 396535 dataset_size: 1047861 - config_name: sk features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 614871 num_examples: 778 download_size: 296352 dataset_size: 614871 - config_name: sr features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 594791 num_examples: 785 download_size: 290084 dataset_size: 594791 - config_name: sv features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 602775 num_examples: 774 download_size: 267273 dataset_size: 602775 - config_name: ta features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1601332 num_examples: 743 download_size: 460555 dataset_size: 1601332 - config_name: te features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 1285217 num_examples: 705 download_size: 406003 dataset_size: 1285217 - config_name: uk features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 981614 num_examples: 770 download_size: 379449 dataset_size: 981614 - config_name: vi features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 778170 num_examples: 785 download_size: 304160 dataset_size: 778170 - config_name: zh features: - name: question dtype: string - name: mc1_targets struct: - name: choices sequence: string - name: labels sequence: int32 - name: mc2_targets struct: - name: choices sequence: string - name: labels sequence: int32 splits: - name: validation num_bytes: 560465 num_examples: 788 download_size: 264083 dataset_size: 560465 configs: - config_name: ar data_files: - split: validation path: ar/validation-* - config_name: bn data_files: - split: validation path: bn/validation-* - config_name: ca data_files: - split: validation path: ca/validation-* - config_name: da data_files: - split: validation path: da/validation-* - config_name: de data_files: - split: validation path: de/validation-* - config_name: es data_files: - split: validation path: es/validation-* - config_name: eu data_files: - split: validation path: eu/validation-* - config_name: fr data_files: - split: validation path: fr/validation-* - config_name: gu data_files: - split: validation path: gu/validation-* - config_name: hi data_files: - split: validation path: hi/validation-* - config_name: hr data_files: - split: validation path: hr/validation-* - config_name: hu data_files: - split: validation path: hu/validation-* - config_name: hy data_files: - split: validation path: hy/validation-* - config_name: id data_files: - split: validation path: id/validation-* - config_name: it data_files: - split: validation path: it/validation-* - config_name: kn data_files: - split: validation path: kn/validation-* - config_name: ml data_files: - split: validation path: ml/validation-* - config_name: mr data_files: - split: validation path: mr/validation-* - config_name: ne data_files: - split: validation path: ne/validation-* - config_name: nl data_files: - split: validation path: nl/validation-* - config_name: pt data_files: - split: validation path: pt/validation-* - config_name: ro data_files: - split: validation path: ro/validation-* - config_name: ru data_files: - split: validation path: ru/validation-* - config_name: sk data_files: - split: validation path: sk/validation-* - config_name: sr data_files: - split: validation path: sr/validation-* - config_name: sv data_files: - split: validation path: sv/validation-* - config_name: ta data_files: - split: validation path: ta/validation-* - config_name: te data_files: - split: validation path: te/validation-* - config_name: uk data_files: - split: validation path: uk/validation-* - config_name: vi data_files: - split: validation path: vi/validation-* - config_name: zh data_files: - split: validation path: zh/validation-* ---该数据集包含多种语言版本，每个版本都包含一个问题特征和两个多选题目标特征。每个多选题目标特征包含选项和标签。数据集主要用于验证集，每个验证集分割信息包括字节数和示例数。

原始信息汇总

数据集概述

数据集配置

该数据集包含多个语言配置，每个配置包含以下信息：

配置列表

• ar (阿拉伯语)
• bn (孟加拉语)
• ca (加泰罗尼亚语)
• da (丹麦语)
• de (德语)
• es (西班牙语)
• eu (巴斯克语)
• fr (法语)
• gu (古吉拉特语)
• hi (印地语)
• hr (克罗地亚语)
• hu (匈牙利语)
• hy (亚美尼亚语)
• id (印度尼西亚语)
• it (意大利语)
• kn (卡纳达语)
• ml (马拉雅拉姆语)
• mr (马拉地语)
• ne (尼泊尔语)
• nl (荷兰语)
• pt (葡萄牙语)
• ro (罗马尼亚语)
• ru (俄语)
• sk (斯洛伐克语)
• sr (塞尔维亚语)
• sv (瑞典语)
• ta (泰米尔语)
• te (泰卢固语)
• uk (乌克兰语)
• vi (越南语)
• zh (中文)

特征结构

每个配置包含以下特征：

• question: 问题，数据类型为 string。
• mc1_targets: 多选目标1，包含以下子结构：
  • choices: 选项，数据类型为 sequence 的 string。
  • labels: 标签，数据类型为 sequence 的 int32。
• mc2_targets: 多选目标2，包含以下子结构：
  • choices: 选项，数据类型为 sequence 的 string。
  • labels: 标签，数据类型为 sequence 的 int32。

数据分割

每个配置包含一个验证集 (validation)，具体信息如下：

• num_bytes: 数据字节数。
• num_examples: 样本数量。

数据文件路径

每个配置的验证集数据文件路径如下：

• ar: ar/validation-*
• bn: bn/validation-*
• ca: ca/validation-*
• da: da/validation-*
• de: de/validation-*
• es: es/validation-*
• eu: eu/validation-*
• fr: fr/validation-*
• gu: gu/validation-*
• hi: hi/validation-*
• hr: hr/validation-*
• hu: hu/validation-*
• hy: hy/validation-*
• id: id/validation-*
• it: it/validation-*
• kn: kn/validation-*
• ml: ml/validation-*
• mr: mr/validation-*
• ne: ne/validation-*
• nl: nl/validation-*
• pt: pt/validation-*
• ro: ro/validation-*
• ru: ru/validation-*
• sk: sk/validation-*
• sr: sr/validation-*
• sv: sv/validation-*
• ta: ta/validation-*
• te: te/validation-*
• uk: uk/validation-*
• vi: vi/validation-*
• zh: zh/validation-*

数据集大小

每个配置的下载大小和数据集大小如下：

配置名称	下载大小 (bytes)	数据集大小 (bytes)
ar	334900	866140
bn	427655	1392045
ca	285221	637317
da	269773	597726
de	297548	673371
es	292004	660712
eu	302334	636227
fr	302523	701928
gu	387499	1204342
hi	415177	1398051
hr	279158	578523
hu	303830	653747
hy	284970	703742
id	266654	650389
it	290933	649613
kn	389616	1232083
ml	450760	1477816
mr	438986	1410185
ne	430638	1396548
nl	284342	643358
pt	287376	647883
ro	292661	655703
ru	396535	1047861
sk	296352	614871
sr	290084	594791
sv	267273	602775
ta	460555	1601332
te	406003	1285217
uk	379449	981614
vi	304160	778170
zh	264083	560465

搜集汇总

数据集介绍

构建方式

在自然语言处理与多语言推理评估的交叉领域中，hynky/okapi_arc_challenge数据集的构建基于广泛认可的ARC Challenge基准，通过将原始英文科学问答题目进行多语言翻译与本地化适配而成。该数据集覆盖了包括阿拉伯语、孟加拉语、加泰罗尼亚语等在内的31种语言，每种语言配置均独立维护，并仅包含验证集划分。每条样本由一个问题字段以及两组多选题答案结构（mc1_targets和mc2_targets）组成，其中每组答案结构包含选项列表与对应的正确标签，从而实现了对模型在不同语言环境下科学推理能力的标准化评测。

使用方法

使用该数据集时，研究者可通过HuggingFace Datasets库按语言配置名（如'ar'、'bn'等）加载对应子集，默认仅提供验证集用于模型评估。加载后的数据可直接用于多选问答任务的推理测试：对于每个样本，模型需从mc1_targets或mc2_targets的候选答案中选择正确选项，并通过与labels字段的比对计算准确率。推荐在评估时同时报告mc1（单一正确）与mc2（多重正确）的得分，以全面衡量模型在复杂推理场景下的表现。该数据集无需额外预处理，可直接作为多语言科学推理评估的标准化基准。

背景与挑战

背景概述

大规模语言模型的推理能力评估是自然语言处理领域的前沿课题，而多语言科学问答基准的构建则成为衡量模型跨语言泛化能力的关键工具。hynky/okapi_arc_challenge数据集由研究团队基于Allen AI的ARC（AI2 Reasoning Challenge）数据集扩展而来，旨在将原本仅覆盖英语的科学推理问题迁移至32种语言，包括阿拉伯语、孟加拉语、中文等低资源语言。该数据集的核心研究问题在于检验多语言模型在复杂科学推理任务中的表现，尤其是面对需要常识与逻辑推导的多选题时，模型能否超越单纯的语言统计模式。通过提供涵盖自然、物理等领域的标准化问答对，该数据集为评估多语言预训练模型的推理鲁棒性提供了重要基准，对推动跨语言人工智能教育应用具有深远影响。

当前挑战

该数据集所解决的领域挑战在于弥合多语言科学推理评估的空白，传统基准如ARC仅覆盖英语，难以反映模型在非英语场景下的真实推理能力。构建过程中面临的主要挑战包括：1）确保不同语言版本间问题语义的严格等价性，避免翻译歧义导致推理难度偏移；2）处理低资源语言（如古吉拉特语、泰卢固语）中科学术语的稀缺性，需依赖人工校验与领域专家介入；3）平衡各语言样本数量（验证集规模在553至789例之间），以维持评估的统计有效性；4）设计多选答案格式（mc1_targets与mc2_targets）以兼容不同推理粒度，同时避免选项排列偏差对模型表现的干扰。

常用场景

经典使用场景

hynky/okapi_arc_challenge 数据集是面向多语言常识推理评测的基准资源，其核心设计源自 Allen Institute for AI 提出的 ARC（AI2 Reasoning Challenge）挑战集，并经由 Okapi 项目扩展至 30 余种语言。该数据集以多项选择题形式呈现，每道题目包含一个自然语言问题及两组候选答案（mc1 与 mc2），分别对应单一正确答案与多重可能正确答案的标注模式。经典使用场景包括评估预训练语言模型在跨语言环境下的知识迁移能力、常识推理稳健性，以及对比不同语种间模型推理性能的差异。研究者可借助该数据集检验模型在低资源语言上的泛化表现，从而揭示多语言表示学习的内在瓶颈。

解决学术问题

该数据集主要解决了多语言常识推理评估中缺乏统一、高质量标注基准的学术难题。传统 ARC 数据集仅覆盖英文，难以衡量模型在非英语环境中的推理深度。hynky/okapi_arc_challenge 通过系统化翻译与人工校验，构建了覆盖印欧语系、汉藏语系、达罗毗荼语系等多样语族的平行评测集，使研究者能够量化语言模型在跨语言场景下的推理一致性。其意义在于为多语言自然语言理解提供了可复现的评估框架，推动了多语言预训练模型（如 mBERT、XLM-R）在常识推理任务上的公平比较，并揭示了当前模型在语系差异与资源丰度影响下的性能衰减规律。

实际应用

在实际应用中，该数据集被广泛用于多语言问答系统、跨语言信息检索以及教育领域的智能辅导工具开发。例如，企业可借助该数据集评估其多语言客服机器人对常识性问题的理解能力，确保不同语言用户获得一致的应答质量。在低资源语言场景下，该数据集帮助开发者识别模型的知识盲区，从而针对性优化训练数据配比或引入跨语言知识蒸馏策略。此外，该数据集还可作为语言教学评估的辅助工具，通过分析模型对不同语种常识问题的回答正确率，间接反映语言资源数字化建设的均衡性。

数据集最近研究

最新研究方向

在自然语言处理领域，多语言常识推理正成为评估大型语言模型跨语言泛化能力的前沿焦点。hynky/okapi_arc_challenge数据集作为ARC Challenge的多语言版本，覆盖阿拉伯语、孟加拉语、中文等30余种语言，为探究模型在低资源语言上的逻辑推理表现提供了关键基准。当前研究热点集中于利用该数据集检测多语言模型在抽象推理与知识迁移中的偏差，尤其是针对印欧语系与非印欧语系之间的性能鸿沟。该数据集的发布不仅推动了多语言AI系统的公平性评估，也为跨文化知识图谱构建与教育智能体开发奠定了实证基础，其影响力正随着全球多语言应用需求的激增而持续扩大。