Opportunity++: A Multimodal Dataset for Video- and Wearable, Object and Ambient Sensors-based Human Activity Recognition

Opportunity++ is a precisely annotated dataset designed to support AI and machine learning research focused on the multimodal perception and learning of human activities (e.g. short actions, gestures, modes of locomotion, higher-level behavior). The Opportunity++ dataset is a significant multimodal extension of the original OPPORTUNITY Activity Recognition Dataset available at https://archive.ics.uci.edu/ml/datasets/OPPORTUNITY+Activity+Recognition. Opportunity++ includes the original video recordings as well as video-derived skeleton tracking data. This enables a wide-range of novel multimodal activity recognition research based on video data, ambient- and object-integrated sensors and wearable sensors (classification, automatic data segmentation, sensor fusion, feature extraction, etc).This release includes:Body-worn sensors: 7 inertial measurement units, 12 3D acceleration sensors, 4 3D localization informationObject sensors: 12 objects with 3D acceleration and 2D rate of turnAmbient sensors: 13 switches and 8 3D acceleration sensorsNewly released anonymized side-view videosNewly released OpenPose tracks for all the people in the videos. This includes the coordinates of the joints (nose, neck, …) of all the users in the video frames.The dataset included data from 4 users performing everyday living activities in a kitchen environment. For each user the dataset includes 6 different runs. Five runs, termed Activity of Daily Living (ADL), followed a given scenario as detailed below. The sixth termed Drill Run, was designed to generate a large number of activity instances in a more constrained scenario. The ADL run consists of temporally unfolding situations. In each situation (e.g. preparing sandwich), a large number of action primitives occur (e.g. reach for bread, move to bread cutter, operate bread cutter).The dataset includes a total of 19.75 hours of sensor data annotated with multiple tracks: 1.88 hours of actions performed with any of the two hands, 6.01 hours of locomotion status, 3.02 hours of annotated data for action performed with a specific hand and 4.89 hours of high level activities. Moreover, the sensors placed on the objects produced a total of 3.92 hours of annotated data.Overall, the dataset comprise of more than 24000 unique annotations, divided in 2551 activity instances for \texttt{ML both arm}, 3653 activity instances of locomotion, 12242 action instances performed with a single specific hand, 122 instances of high level activities and 6103 instances of interaction with the object in the kitchen.Opportunity++是一款经精准标注的数据集，旨在支撑面向人类活动多模态感知与学习的人工智能(Artificial Intelligence, AI)与机器学习研究，涵盖各类人类行为（如短时动作、手势、移动模式、高阶行为等）。Opportunity++数据集是原始OPPORTUNITY活动识别数据集的重要多模态扩展版本，原始数据集可从https://archive.ics.uci.edu/ml/datasets/OPPORTUNITY+Activity+Recognition获取。Opportunity++保留了原始视频录制内容，同时新增了源自视频的骨骼追踪数据，这为基于视频数据、环境与物体集成传感器以及可穿戴传感器的各类新型多模态活动识别研究提供了支撑，研究方向涵盖分类、自动数据分割、传感器融合、特征提取等。 本次发布包含以下内容： 可穿戴传感器：7个惯性测量单元、12个3D加速度传感器、4组3D定位信息； 物体传感器：12个配备3D加速度传感器与2D角速度传感器的物体； 环境传感器：13个开关传感器与8个3D加速度传感器； 全新发布的匿名侧视角视频； 针对视频中所有人物的OpenPose追踪数据，包含视频帧中所有受试者的关节（鼻尖、颈部等）坐标信息。 该数据集收录了4名受试者在厨房环境中完成日常起居活动的相关数据。每名受试者对应6组不同的录制运行：其中5组为日常生活活动(Activity of Daily Living, ADL)，遵循如下预设场景；第6组为演练运行(Drill Run)，旨在在更受限的场景中生成大量活动样本。日常生活活动录制运行包含随时间展开的多种场景，在每个场景（如制作三明治）中，会出现大量动作原语（如伸手取面包、移动至面包切片机旁、操作面包切片机等）。 该数据集总计包含19.75小时的标注多轨道传感器数据：其中1.88小时为双手任意一手完成的动作标注数据，6.01小时为移动状态标注数据，3.02小时为特定单手握持动作的标注数据，4.89小时为高阶活动标注数据。此外，部署在物体上的传感器总计产生了3.92小时的标注数据。 总体而言，该数据集包含超过24000条唯一标注，具体划分为：2551条双臂活动实例、3653条移动活动实例、12242条特定单手动作实例、122条高阶活动实例，以及6103条与厨房内物体交互的活动实例。