SMART-101

该数据集名为SMART-101，专为6至8岁儿童设计，包含101个独特的视觉-语言谜题。每个谜题都有一张图片和一个问题，解答这些问题需要混合运用基础的算术、代数和空间推理等技能。此外，该数据集还包括四种不同的数据划分方式（实例划分、答案划分、谜题划分和少样本划分），以评估算法在不同方面的泛化能力。规模上，共有101个独特的谜题，并为每个谜题程序化生成了实例。该数据集的任务是评估神经网络在解决视觉-语言谜题时在抽象、推理和泛化能力方面的表现。This dataset, named SMART-101, is designed for children aged 6 to 8 and contains 101 unique visual-language puzzles. Each puzzle comprises an image and a question, and solving these puzzles requires the integrated application of skills including basic arithmetic, algebra, and spatial reasoning. Furthermore, this dataset provides four distinct data splitting approaches: instance split, answer split, puzzle split, and few-shot split, to evaluate the generalization capability of algorithms across different dimensions. In terms of scale, there are 101 unique puzzles, with instances programmatically generated for each one. The core task of this dataset is to evaluate the performance of neural networks in terms of abstraction, reasoning, and generalization abilities when solving visual-language puzzles.

搜集汇总

背景与挑战

背景概述

SMART-101是一个多模态算法推理任务数据集，旨在评估神经网络在解决儿童视觉语言谜题中的抽象、演绎和泛化能力。它包含101个独特谜题，每个谜题结合图片和问题，需要混合算术、代数等基本技能，并通过程序化生成新实例来扩展训练数据。实验表明，尽管现有模型在监督设置下表现尚可，但在泛化能力上不及人类儿童，且大型语言模型常给出错误答案。