Verifee

我们介绍了Verifee数据集: 具有细粒度的可信赖性注释的新闻文章的新颖数据集。我们开发了一种详细的方法，根据文本的参数评估文本，包括编辑透明度，记者惯例和客观报道，同时惩罚操纵技术。我们带来了来自社交，媒体和计算机科学的各种研究人员，以克服这一跨学科问题的障碍和有限的框架。我们从近60个捷克在线新闻来源收集了超过10,000篇独特的文章。这些被归类为我们提出的信誉范围内的4个类别之一，从完全值得信赖的文章一直到操纵性文章。我们会提供详细的统计数据，并研究整个集合中出现的趋势。最后，我们在可信性分类任务上使用我们的数据集微调多个流行的序列到序列语言模型，并报告0.52的最佳测试F-1得分。我们在https:// verifee.ai/research上全文开源数据集，注释方法和注释的说明，以实现轻松的构建工作。我们相信类似的方法可以帮助防止虚假信息，并在媒体素养领域进行教育。

We introduce the Verifee Dataset: a novel dataset of news articles with fine-grained trustworthiness annotations. We developed a rigorous methodology to evaluate news texts against multiple dimensions including editorial transparency, journalistic conventions, and objective reporting, while penalizing manipulative writing tactics. We assembled a diverse team of researchers from the fields of communication, media studies, and computer science to address the interdisciplinary barriers and underdeveloped existing frameworks surrounding this topic. We collected over 10,000 unique news articles from nearly 60 Czech online news outlets. These articles were categorized into one of four classes within our proposed trustworthiness spectrum, ranging from fully trustworthy pieces to highly manipulative articles. We provide detailed statistical analyses and investigate emerging trends across the entire dataset. Finally, we fine-tuned several popular sequence-to-sequence large language models (LLMs) on our dataset for the news trustworthiness classification task, and report a best test F1-score of 0.52. We fully open-source the dataset, annotation methodology, and annotation guidelines at https://verifee.ai/research to facilitate straightforward adoption and reproducible research. We believe that similar methodologies can aid in combating misinformation and advancing media literacy education.

搜集汇总

背景与挑战

背景概述

Verifee是一个包含细粒度可信赖性注释的新闻文章数据集，收录了超过10,000篇来自捷克在线新闻源的文章，并根据编辑透明度、记者惯例和客观报道等参数分为4个信誉类别。该数据集可用于可信性分类任务，在微调序列到序列模型上取得了0.52的F-1得分，由伦敦大学学院和查理大学于2022年发布。