eStonefish-Scenes: A Sim-to-Real Validated and Robot-Centric Event-based Optical Flow Dataset for Underwater Vehicles (Synthetic)

eStonefish-Scenes is a large-scale synthetic dataset designed to bridge the gap between simulation and reality for underwater event-based vision. Generated using the Stonefish simulator, this dataset provides high-quality event streams, grayscale images, and dense ground-truth optical flow tailored for Autonomous Underwater Vehicles (AUVs).

Features

Diverse Environments: Includes both Rocky (barren seabed) and Reef environments, the latter populated with procedurally generated coral clusters to create texture-rich scenes.

Dynamic Scenarios: Features static scenes for geometric learning and dynamic scenes containing schools of fish, simulated using the Stonefish-Boids package to mimic realistic behaviors like obstacle avoidance and cohesion.

Robot-Centric Acquisition: Data is captured from a simulated BlueROV2 performing 6-DOF maneuvers (heave, surge, sway, yaw) with both down-looking and forward-looking camera configurations.

Optimized Format: Stored in the HDF5-based eWiz format, ensuring efficient storage and access.

Usage

To utilize this dataset, we recommend using the eWiz library. eWiz is a comprehensive toolkit developed alongside this dataset that handles data loading, visualization, event representation encoding, and augmentation.

Library Repository: https://github.com/CIRS-Girona/ewiz

Scene Generation Pipeline: https://github.com/CIRS-Girona/stonefish-scenegen

Data Generation Pipeline

The dataset was produced using a modular, open-source pipeline built upon the Stonefish simulator. The process consists of three main components:

Stonefish-SceneGen: A procedural generation tool that automates the creation of diverse, texture-rich underwater environments by randomly populating the seabed with various coral clusters and flora (you need to create your own coral and flora models).

Stonefish-Boids: A package that enhances scene realism by simulating schools of fish using the Boids algorithm. It models natural behaviors, alignment, cohesion, and separation, and incorporates real-time obstacle avoidance using Octree-based environment mapping.

eStonefish-Scenes: The primary ROS-based collection interface that manages the simulated BlueROV2 and sensors. It records synchronized event streams, grayscale imagery, and noise-free ground truth optical flow, which are subsequently converted into the optimized eWiz HDF5 format for efficient processing.

The complete generation pipeline is publicly available to allow users to create custom datasets with varying environmental parameters, it is available on: https://github.com/CIRS-Girona/estonefish-scenes

Data Structure

The data is stored in the optimized eWiz HDF5 format, which supports efficient slicing and low-memory access.

events.hdf5: Compressed event streams (x, y, t, p).

gray.hdf5: Synchronized grayscale frames.

flow.hdf5: Dense optical flow ground truth.

props.json: Dataset properties and metadata.

Dataset Update

This dataset is provided at the native resolution of the DAVIS346 sensor ($346 \times 260$ pixels), a design choice specifically intended to facilitate the training of lightweight Convolutional Neural Networks (CNNs) and Spiking Neural Networks (SNNs). By maintaining a lower spatial resolution, we significantly reduce the computational complexity and memory footprint required for processing, enabling faster training iterations and allowing researchers to develop models that are deployable on the resource-constrained embedded hardware typically found on Autonomous Underwater Vehicles (AUVs). This focus on efficiency ensures that developed algorithms can achieve the low latency required for real-time tasks like obstacle avoidance and visual odometry, without the computational overhead associated with high-resolution imagery.

Validation

This synthetic dataset has been validated against real-world underwater data. For the corresponding real-world validation dataset, please refer to the real validation dataset.

Inquiries & Support

For any questions regarding the eStonefish-Scenes dataset, or the real-world validation data, or the data generation pipeline, please contact the corresponding author:

Jad Mansour | Email: [e-mail]

We hope that this work encourages the community to dive deeper into event-based underwater perception, preferably with fewer leaks in the pipeline and more flow in the right direction!