BBAI Dataset (Black-box Agent Integration)

该数据集用于评估 Black-box Multi-agent Integration 的任务，该任务侧重于大规模组合多个黑盒会话代理的功能。它提供了探索两个主要探索框架的数据：问题代理配对和问题响应配对。总体而言，该数据集包含 5550 个话语，每个问题有 19 个问答对（来自 19 个代理中的每个），在 37 个域中总共有 105,450 个。话语分为 3700 个话语（每个域 100 个示例）用于训练集和 1850 个话语（每个域 50 个）用于测试集。训练集和测试集分别包含 2399 和 1186 个话语，其中至少有一个积极的问答对。在其余示例中，没有一个代理能够达到注释者协议 (>= 3)。This dataset is designed to evaluate the Black-box Multi-agent Integration task, which focuses on large-scale combination of functionalities from multiple black-box conversational agents. It provides data for exploring two primary research frameworks: question-agent pairing and question-response pairing. Overall, this dataset contains 5,550 utterances, with each question paired with 19 question-answer pairs (one from each of the 19 agents), totaling 105,450 instances across 37 domains. The utterances are split into a training set with 3,700 utterances (100 examples per domain) and a test set with 1,850 utterances (50 examples per domain). The training and test sets respectively contain 2,399 and 1,186 utterances that have at least one positive question-answer pair. For the remaining instances, none of the agents achieved an inter-annotator agreement score of ≥3.

搜集汇总

背景与挑战

背景概述

BBAI数据集用于评估黑盒多智能体会话代理的集成任务，包含5550个话语，每个问题对应19个代理的问答对，总计105,450个数据，覆盖37个域，并划分为训练集和测试集。该数据集由密歇根大学、福特汽车公司和范德比尔特大学于2022年发布。