Video Scene Parsing in the Wild

We present a new dataset with the target of advancing the scene parsing task from images to videos. Our dataset aims to perform Video Scene Parsing in the Wild (VSPW), which covers a wide range of real-world scenarios and categories. To be specific, our VSPW is featured from the following aspects: 1) Well-trimmed longtemporal clips. Each video contains a complete shot, lasting around 5 seconds on average. 2) Dense annotation. The pixel-level annotations are provided at a high frame rate of 15 f/s. 3) High resolution. Over 96% of the captured videos are with high spatial resolutions from 720P to 4K. We totally annotate 3,536 videos, including 251,633 frames from 124 categories. To the best of our knowledge, our VSPW is the first attempt to tackle the challenging video scene parsing task in the wild by considering diverse scenarios. Based on VSPW, we design a generic Temporal Context Blending (TCB) network, which can effectively harness long-range contextual information from the past frames to help segment the current one. Extensive experiments show that our TCB network improves both the segmentation performance and temporal stability comparing with image-/video-based state-of-the-art methods. We hope that the scale, diversity, long-temporal, and high frame rate of our VSPW can significantly advance the research of video scene parsing and beyond.

本研究提出一款全新数据集，旨在将场景解析任务从图像领域拓展至视频领域。本数据集致力于完成野外视频场景解析（Video Scene Parsing in the Wild，VSPW）任务，涵盖丰富的真实应用场景与类别。具体而言，该VSPW数据集具备以下三大核心特性：1. 精准剪辑的长时序片段：每段视频均包含完整镜头，平均时长约5秒；2. 稠密标注：以15帧每秒（f/s）的高帧率提供像素级标注；3. 高分辨率特性：采集的视频中超过96%具备720P至4K的高空间分辨率。本次标注共完成3536段视频的标注，总计包含251633帧图像，覆盖124个类别。据我们所知，本VSPW数据集是首个针对多样化真实场景下极具挑战性的野外视频场景解析任务的尝试。基于该VSPW数据集，我们设计了一款通用的时序上下文融合（Temporal Context Blending，TCB）网络，可有效利用过往帧的长程上下文信息，辅助当前帧的语义分割任务。大量对比实验表明，相较于当前基于图像或视频的现有最优方法，本TCB网络在分割性能与时序稳定性上均实现了显著提升。我们期望本VSPW数据集的大规模性、类别多样性、长时序特性与高帧率标注等优势，能够有力推动野外视频场景解析乃至相关领域的研究进展。

搜集汇总

背景与挑战

背景概述

VSPW是一个大规模视频场景解析数据集，包含3,536个高分辨率视频（720P至4K）和251,633帧的密集像素级标注，覆盖124个类别。该数据集以长时视频片段（平均5秒）和高帧率标注（15帧/秒）为特点，旨在推动真实场景下的视频解析研究。