CALM

CALM是一个综合评估语言模型偏差的多任务基准数据集，由宾夕法尼亚州立大学计算机科学与工程系的研究团队创建。该数据集整合了16个现有数据集，用于问题回答、情感分析和自然语言推理三个NLP任务，共包含78,400个提示，用于测量性别和种族偏差。CALM通过筛选224个模板，并结合350个代表不同社会群体的常用人名，生成提示，旨在解决语言模型在处理不同社会群体时的性能差异问题。CALM is a multi-task benchmark dataset for comprehensive evaluation of language model biases, created by a research team from the Department of Computer Science and Engineering at Pennsylvania State University. This dataset incorporates 16 existing datasets across three NLP tasks: question answering, sentiment analysis, and natural language inference, and contains a total of 78,400 prompts designed to measure gender and racial biases. CALM generates prompts by screening 224 templates and combining 350 common proper nouns representing diverse social groups, with the objective of resolving performance disparities of language models when processing different social groups.

搜集汇总

背景与挑战

背景概述

CALM是一个用于全面评估语言模型偏见的多任务基准数据集，旨在通过多种任务测试模型偏见。数据集包含代码和模板生成工具，支持使用不同国家的人名扩展，并提供了在Hugging Face上的公开访问链接。