Referring Expressions for DAVIS 2016 & 2017

我们的任务是在给定语言引用表达式的情况下，在所有视频帧上本地化并提供对象的像素级掩码，该语言引用表达式通过仅查看第一帧或完整视频获得。为了验证我们的方法，我们采用了两个流行的视频对象分割数据集，DAVIS16 [38] 和 DAVIS17 [42]。这两个数据集引入了各种挑战，包含具有单个或多个显着对象、拥挤场景、相似外观实例、遮挡、相机视图变化、快速运动等的视频。DAVIS16 [38] 包含 30 个训练视频和 20 个不同对象类别的测试视频所有帧都以像素级精度进行注释。请注意，在此数据集中，每个视频仅注释一个对象。对于多对象视频分割任务，我们考虑 DAVIS17。与 DAVIS16 相比，这是一个更具挑战性的数据集，每个视频有多个对象注释，并且场景更复杂，干扰因素、遮挡、对象更小和结构更精细。总体而言，DAVIS17 由一个包含 60 个视频的训练集和一个包含 30 个序列的验证/测试开发/测试挑战集组成。由于我们的目标是使用语言规范对视频中的对象进行分割，因此我们在 DAVIS16 和 DAVIS17 中使用无歧义的引用表达式来扩充所有使用掩码标签注释的对象。我们遵循 [34] 的工作，并要求注释者通过仅查看视频的第一帧来提供对象的语言描述，该对象具有掩码注释。然后另一个注释器被赋予第一帧和相应的描述，并被要求识别所引用的对象。如果注释者无法正确识别对象，则更正描述以消除歧义并唯一地指定对象。我们收集了由非计算机视觉专家注释的每个目标对象的两个引用表达式（注释器 1、2）。但是，通过仅查看第一帧，获得的参考表达可能对整个视频无效。 （我们实际上量化了，收集到的描述中只有 15% 会随着时间的推移变得无效，并且它不会强烈影响分割结果，因为时间一致性步骤有助于消除一些此类情况的歧义，请参阅补充材料以获取详细信息。）此外，在许多应用程序，例如视频编辑或基于视频的广告，用户可以访问完整的视频。提供对所有帧都有效的语言查询可能会减少编辑时间并产生更连贯的预测。因此，在 DAVIS17 上，我们要求工作人员通过查看完整视频来提供对象的描述。我们为每个目标对象收集了一个完整视频类型的表达式。未来的工作可能会选择使用任一设置。第一帧/完整视频表达的平均长度为 5.5/6.3 个单词。对于 DAVIS17 第一帧注释，我们注意到注释器 1 给出的描述比注释器 2 给出的描述长（6.4 对 4.6 个字）。我们在§5 中评估了描述长度对接地性能的影响。此外，与完整视频相关的表达比第一帧描述更频繁地提及动词（44% 对 25%）。这是直观的，因为提及随时间改变其外观和位置的对象可能需要提及其动作。所有注释中超过 50% 存在形容词。其中大多数是指颜色（超过 70%）、形状和大小（7%）以及空间/排序词（6% 第一帧与 13% 完整视频表达）。完整的视频表达也有更多的副词和介词，总体上比为第一帧提供的更复杂。总体增强的 DAVIS16/17 包含 1.2k 个引用表达式，用于 150 个视频和 10k 帧的 400 多个对象。我们相信收集到的数据将对分割以及视觉和语言社区感兴趣，从而为探索语言作为视频对象分割的替代输入提供了机会。

Our task is to localize objects across all video frames and provide their pixel-level masks given a language referring expression, which can be obtained by only viewing the first frame or the full video. To validate our method, we adopt two popular video object segmentation datasets: DAVIS16 [38] and DAVIS17 [42]. These two datasets introduce diverse challenges, including videos with single or multiple salient objects, crowded scenes, visually similar instances, occlusions, camera view changes, fast motions, and more. DAVIS16 [38] contains 30 training videos and test videos for 20 distinct object categories, with all frames annotated at pixel-level precision. Note that in this dataset, only one object is annotated per video. For the multi-object video segmentation task, we consider DAVIS17. Compared to DAVIS16, this is a more challenging dataset, with multiple object annotations per video, more complex scenes, distractions, occlusions, smaller objects, and finer-grained structures. Overall, DAVIS17 consists of a training set with 60 videos and a validation/test development/test challenge set with 30 sequences. Since our goal is to segment objects in videos using language specifications, we augment all objects annotated with mask labels in DAVIS16 and DAVIS17 with unambiguous referring expressions. We follow the work of [34] and ask annotators to provide language descriptions of the object with mask annotations by only viewing the first frame of the video. Then another annotator is given the first frame and the corresponding description, and is asked to identify the referenced object. If the annotator cannot correctly recognize the object, the description is revised to eliminate ambiguity and uniquely specify the object. We collected two referring expressions (from Annotator 1 and Annotator 2) for each target object, where the annotators were non-computer vision experts. However, referring expressions obtained by only viewing the first frame may be invalid for the entire video. (We actually quantified that only 15% of the collected descriptions become invalid over time, and this does not strongly impact segmentation results, as the temporal consistency step helps disambiguate some of these cases; see the supplementary material for details.) Furthermore, in many applications such as video editing or video-based advertising, users have access to the full video. Providing language queries that are valid for all frames may reduce editing time and yield more coherent predictions. Therefore, for DAVIS17, we ask workers to provide descriptions of objects by viewing the full video. We collected one full-video-style expression for each target object. Future work may choose to use either setting. The average length of the first-frame/full-video expressions is 5.5/6.3 words. For DAVIS17 first-frame annotations, we note that descriptions given by Annotator 1 are longer than those from Annotator 2 (6.4 vs. 4.6 words). We evaluate the impact of description length on grounding performance in §5. Additionally, expressions associated with full videos mention verbs more frequently than first-frame descriptions (44% vs. 25%). This is intuitive, as referring to objects that change their appearance and position over time may require mentioning their actions. Over 50% of all annotations contain adjectives. Most of these refer to color (over 70%), shape and size (7%), and spatial/ordering words (6% for first-frame vs. 13% for full-video expressions). Full-video expressions also contain more adverbs and prepositions, and are overall more complex than those provided for the first frame. Overall, the augmented DAVIS16/17 contains 1.2k referring expressions for over 400 objects across 150 videos and 10k frames. We believe the collected data will be of interest to the segmentation, vision, and language communities, providing opportunities to explore language as an alternative input for video object segmentation.

搜集汇总

背景与挑战

背景概述

该数据集是DAVIS16和DAVIS17视频对象分割数据集的扩展版本，通过为每个标注对象添加语言引用表达式（平均长度5.5-6.3个单词），支持基于语言描述的视频对象分割任务。数据集包含150个视频、10k帧和400多个对象的1.2k条语言标注，涵盖单对象和多对象复杂场景。