CRMArena
收藏资源简介:
CRMArena是一个用于评估大型语言模型(LLM)代理在现实环境中执行实际工作任务能力的基准。该基准包含16个常用的工业对象,这些对象具有高度的互联性,并包含潜在变量以模拟现实数据分布。每个实例包含标准答案、任务名称、查询/任务的元数据和LLM代理应响应的查询。
CRMArena is a benchmark for evaluating the capability of Large Language Model (LLM) agents to perform real-world operational tasks in realistic environments. This benchmark includes 16 commonly used industrial objects, which are highly interconnected and incorporate latent variables to simulate real-world data distributions. Each instance contains the standard answer, task name, metadata of the query/task, and the query that the LLM agent should respond to.
CRMArena 数据集概述
基本信息
- 语言: 英语
- 许可证: Apache 2.0
- 多语言性: 单语种
- 数据规模: 1K<n<10K
- 标签:
- agent
- LLM
- work-benchmark
- evaluation
数据集描述
CRMArena 是一个用于评估大型语言模型(LLM)代理在现实环境中执行实际工作任务能力的基准。该基准包含16个常用的工业对象(如账户、订单、知识文章、案例),并具有高度的互联性,以及潜在变量(如投诉习惯、政策违规),以模拟现实数据分布。
数据字段
answer: 标准答案task: 任务名称metadata: 查询/任务的元数据,应作为系统提示的一部分query: LLM 代理应响应的查询
相关论文
- 论文标题: CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
- 论文链接: https://arxiv.org/abs/2411.02305
- 代码库: https://github.com/SalesforceAIResearch/CRMArena/
- 排行榜: https://huggingface.co/spaces/Salesforce/CRMArena-Leaderboard
引用
@misc{huang-etal-2024-crmarena, title = "CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments", author = "Huang, Kung-Hsiang and Prabhakar, Akshara and Dhawan, Sidharth and Mao, Yixin and Wang, Huan and Savarese, Silvio and Xiong, Caiming and Laban, Philippe and Wu, Chien-Sheng", year = "2024", archivePrefix = "arXiv", eprint={2411.02305}, primaryClass={cs.CL} }




