kyutai/KairosQA

KairosQA是一个基于时间基础的问答数据集，旨在评估大型语言模型（LLMs）的时间对齐和推理能力。与静态基准不同，该数据集专注于随时间变化的事实，特别是来自Wikidata的主语-关系-宾语三元组，这些三元组在2018年至2025年之间至少发生了两次变化（如相关论文所述）。数据集优先考虑流行主题（基于维基百科页面浏览量），以确保评估失败是由于时间错位而非简单缺乏世界知识。它最初设计用于评估按时间顺序排列的数据预训练与随机预训练数据相比在新鲜度方面的改进。数据集来源为Wikidata（经过时间变化过滤），包含7,167个主题-关系对，重点领域包括体育、组织、职业相关事实和事件。格式为通过GPT-4o mini生成的多项选择题，时间范围涵盖2014年至2025年的评估快照。

KairosQA is a time-grounded question answering dataset designed to evaluate the temporal alignment and reasoning capabilities of large language models (LLMs). Unlike static benchmarks, this dataset focuses on time-evolving facts, specifically subject-relation-object triples from Wikidata that have undergone at least two changes between 2018 and 2025 as described in relevant literature. The dataset prioritizes popular topics based on Wikipedia page views, ensuring that any evaluation failures arise from temporal misalignment rather than a mere lack of world knowledge. It was originally developed to assess improvements in factual freshness when using temporally ordered data for pre-training, compared to pre-training with randomly ordered data. The dataset is sourced from Wikidata filtered for temporal variations, containing 7,167 topic-relation pairs, with core domains including sports, organizations, career-related facts and events. It is formatted as multiple-choice questions generated via GPT-4o mini, with evaluation snapshots spanning the time range from 2014 to 2025.