WSAG

我们提出了一个新的具有挑战性的接地任务: 弱监督的时间文章接地 (WSAG)。具体来说，给定一篇文章和相关视频，WSAG旨在将所有 “基础” 句子本地化到视频中，这些句子可能处于不同的语义范围。因此，我们收集了第一个WSAG数据集来促进这项任务: YouwikiHow，它借用了wikiHow文章和大量YouTube视频中固有的多尺度描述。此外，我们为WSAG提出了一种简单但有效的DualMIL方法，该方法由两级MIL损失和单/跨句约束损失组成。这些培训目标是为这些宽松的假设而精心设计的。We introduce a novel and challenging grounding task: Weakly-Supervised Temporal Article Grounding (WSAG). Specifically, given an article and its associated video, WSAG aims to localize all "foundational" sentences within the video, where these sentences may cover diverse semantic scopes. To facilitate this task, we collect the first WSAG dataset: YouwikiHow, which leverages the inherently multi-scale descriptions from wikiHow articles and a large corpus of YouTube videos. Furthermore, we propose a simple yet effective DualMIL method for WSAG, which comprises two-level MIL losses and single- and cross-sentence constraint losses. These training objectives are carefully tailored for these relaxed assumptions.

搜集汇总

背景与挑战

背景概述

WSAG是一个弱监督时间文章接地任务的数据集，包含wikiHow文章和YouTube视频，旨在将文章句子与视频时间点对应。由哥伦比亚大学等机构于2022年发布，提供了DualMIL方法处理该任务。