ego-1k

# [Ego-1K — A Large-Scale Multiview Video Dataset for Egocentric Vision](https://arxiv.org/abs/2603.13741)

Jae Yong Lee, Daniel Scharstein, Akash Bapat, Hao Hu, Andrew Fu, Haoru Zhao, Paul Sammut, Xiang Li,
Stephen Jeapes, Anik Gupta, Lior David, Saketh Madhuvarasu, Jay Girish Joshi, and Jason Wither

[arxiv.org/abs/2603.13741](https://arxiv.org/abs/2603.13741); to appear in CVPR 2026

We present Ego-1K, a large-scale collection of time-synchronized egocentric multiview videos designed to advance neural 3D video
synthesis and dynamic scene understanding. The dataset contains 956 short (6.7-9.7s) egocentric videos taken with a custom rig with
12 synchronous cameras surrounding a VR headset worn by the user, for a total of 491K frames and 5.9M images. Scene content focuses on hand
motions and hand-object interactions in different settings. Our dataset enables new ways to benchmark egocentric scene reconstruction methods,
and presents unique challenges for existing 3D and 4D novel view synthesis methods due to high disparities and image motion caused by close
dynamic objects and rig egomotion.


## License

[FAIR Noncommercial Research License](https://huggingface.co/facebook/fair-noncommercial-research-license/blob/main/LICENSE)

## Getting Started

See [`quickstart.ipynb`](quickstart.ipynb) for a runnable walkthrough that loads metadata, streams images, and visualizes a 12-camera multiview frame.

## Key Statistics

| Property | Value |
|----------|-------|
| Total recordings | 956 |
| Train split | 860 recordings |
| Test split | 96 recordings |
| Duration per recording | 6.7-9.7 seconds at 60 Hz |
| Frames per recording | 404-583 (mean 514, median 530) |
| Cameras | 12 (6 rectified stereo pairs) |
| Image resolution | 1280 x 1280 pixels (rectified pinhole, 120 deg HFOV) |
| Total frames | 490,966 |
| Total images | 5,891,592 |
| Total duration | 2.3 hours |
| Total shards | 10,216 |
| Total dataset size | ~18 TB (WebDataset tar shards) |
| Typical shard size | ~1.5 GB |


## Dataset Structure

```text
ego-1k/
├── data/
│ ├── train-<scene_id>.parquet # Per-scene metadata index
│ └── test-<scene_id>.parquet
└── shards/
├── train/
│ └── <scene_id>/
│ ├── <scene_id>-0000.tar # WebDataset tar shards (~1.5 GB each)
│ ├── <scene_id>-0001.tar
│ └── ...
└── test/
└── <scene_id>/
└── <scene_id>-0000.tar
```

### Tar Shard Contents

Each tar sample represents one frame across all 12 cameras:

```text
<scene_id>/<frame_id:06d>.200-1.png # Raw PNG bytes (1280x1280)
<scene_id>/<frame_id:06d>.200-2.png
...
<scene_id>/<frame_id:06d>.200-12.png
<scene_id>/<frame_id:06d>.metadata.json # Pose, rig calibration, scene info
```

The `metadata.json` per sample contains:

| Field | Type | Description |
|-------|------|-------------|
| `scene_id` | string | Recording identifier |
| `frame_id` | int | Frame index (0-indexed) |
| `timestamp_ns` | int | Frame timestamp in nanoseconds |
| `pose` | list | 4x4 device-to-world transform (key absent if unavailable) |
| `rig_calibration` | object | Per-camera intrinsics (`K`) and extrinsics (`E`) |
| `source` | string | Capture campaign: `OVD_M1` (lab, 513 recordings), `OVD_M2` (apartment, 414), `DD4` (29) |
| `lux_bins` | string | Lighting level: `51-75`, `76-100`, `101-200`, `201-400`, `401-1000`, `1001+` |
| `tags` | list | Scene diversity tags |

## Parquet Schema

Each row represents a single frame (one timestamp across all 12 cameras):

| Column | Type | Description |
|--------|------|-------------|
| `scene_id` | string | Recording identifier |
| `frame_id` | int32 | Frame index within the recording (0-indexed; number of frames varies per scene, range 404-583) |
| `timestamp_ns` | int64 | Frame timestamp in nanoseconds |
| `source` | string | Capture campaign: `OVD_M1` (lab), `OVD_M2` (apartment), `DD4` |
| `lux_bins` | string | Lighting level: `51-75`, `76-100`, `101-200`, `201-400`, `401-1000`, `1001+` |
| `tags` | string | JSON list of scene diversity tags (85 unique tags covering garments, furnishings, lighting, pose, objects) |
| `shard_name` | string | Relative path to the tar shard containing this frame's images (e.g., `shards/train/<scene_id>/<scene_id>-0002.tar`) |
| `pose` | string | JSON: 4x4 device-to-world transform matrix for this frame (null if pose unavailable) |
| `rig_calibration` | string | JSON: per-camera intrinsics (`K`: 3x3) and extrinsics (`E`: 4x4), static per scene (repeated for each frame for convenience) |

### Calibration Details

The `rig_calibration` column contains a JSON object keyed by camera name (`200-1` through `200-12`), each with:
- **`K`**: 3x3 intrinsic matrix (rectified pinhole projection, 120 deg horizontal FOV)
- **`E`**: 4x4 extrinsic matrix (camera-to-device transform)

The `pose` column contains the 4x4 device-to-world transform, which changes per frame as the headset moves.

## Usage

`load_dataset` returns frame-level **metadata only** (poses, calibration, scene info). Images are stored in WebDataset tar shards — use the `webdataset` library to stream them. See [`quickstart.ipynb`](quickstart.ipynb) for a full working example.

### WebDataset (Recommended for Training)

Stream tar shards for high-throughput sequential access — no per-file API calls. See the notebook for the full `decode_sample` implementation. To wrap it in a PyTorch DataLoader:

```python
dataset = wds.WebDataset(shard_urls, nodesplitter=wds.split_by_node, shardshuffle=True).map(decode_sample)
loader = torch.utils.data.DataLoader(dataset, batch_size=4, num_workers=4)

for batch in loader:
images = batch["images"] # (B, N_cams, 3, 1280, 1280)
break
```

### Parquet Metadata (Random Access)

The Parquet files contain frame-level metadata only (poses, calibration, scene info) — images are stored in the tar shards. Use the `shard_name` column to locate which tar file contains a given frame's images.

```python
shard_url = f"https://huggingface.co/datasets/facebook/ego-1k/resolve/main/{example['shard_name']}"
```

## Citation

```bibtex
@inproceedings{ego1k2026,
title={{Ego-1K}: A Large-Scale Multiview Video Dataset for Egocentric Vision},
author={Jae Yong Lee and Daniel Scharstein and Akash Bapat and Hao Hu and Andrew Fu and Haoru Zhao and Paul Sammut and Xiang Li and Stephen Jeapes and Anik Gupta and Lior David and Saketh Madhuvarasu and Jay Girish Joshi and Jason Wither},
booktitle={CVPR},
year={2026}
}
```

# [Ego-1K — 面向第一人称视觉（Egocentric Vision）的大规模多视图视频数据集](https://arxiv.org/abs/2603.13741)

Jae Yong Lee, Daniel Scharstein, Akash Bapat, Hao Hu, Andrew Fu, Haoru Zhao, Paul Sammut, Xiang Li, Stephen Jeapes, Anik Gupta, Lior David, Saketh Madhuvarasu, Jay Girish Joshi, and Jason Wither

[arxiv.org/abs/2603.13741](https://arxiv.org/abs/2603.13741); 将于CVPR 2026发表

本文提出Ego-1K数据集，这是一个经时间同步的大规模第一人称视觉多视图视频集合，旨在推动神经三维视频合成与动态场景理解领域的发展。该数据集包含956段时长6.7至9.7秒的第一人称视频，采集时用户佩戴VR头戴设备（VR Headset），配套的定制采集装置搭载12台同步相机环绕设备布置，总计包含49.1万帧画面与590万张图像。场景内容聚焦不同环境下的手部动作与手-物体交互行为。本数据集为第一人称场景重建方法提供了全新的基准测试途径，同时由于近距离动态物体与采集装置自身运动带来的高视差与图像运动，也为现有三维、四维新视角合成方法带来了独特挑战。


## 许可协议

[FAIR非商业研究许可协议](https://huggingface.co/facebook/fair-noncommercial-research-license/blob/main/LICENSE)

## 快速入门

请参阅`quickstart.ipynb`文件，其中提供了可运行的完整流程，可加载元数据、流式读取图像并可视化12相机多视图画面。


## 核心统计数据

| 属性 | 数值 |
|----------|-------|
| 总录制段数 | 956 |
| 训练划分 | 860段录制 |
| 测试划分 | 96段录制 |
| 单段录制时长 | 6.7-9.7秒，帧率60 Hz |
| 单段录制帧数 | 404-583帧（平均514帧，中位数530帧） |
| 相机数量 | 12台（6对校正立体相机） |
| 图像分辨率 | 1280 × 1280像素（校正针孔相机，水平视场角120°） |
| 总帧数 | 490,966 |
| 总图像数 | 5,891,592 |
| 总时长 | 2.3小时 |
| 总分片数 | 10,216 |
| 总数据集大小 | ~18 TB（WebDataset（WebDataset）格式tar分片） |
| 典型分片大小 | ~1.5 GB |

## 数据集结构

text
ego-1k/
├── data/
│ ├── train-<scene_id>.parquet # 单场景元数据索引
│ └── test-<scene_id>.parquet
└── shards/
├── train/
│ └── <scene_id>/
│ ├── <scene_id>-0000.tar # WebDataset格式tar分片（单分片约1.5 GB）
│ ├── <scene_id>-0001.tar
│ └── ...
└── test/
└── <scene_id>/
└── <scene_id>-0000.tar

### Tar分片内容

每个tar样本对应所有12台相机的单帧数据：

text
<scene_id>/<frame_id:06d>.200-1.png # 原始PNG字节数据（1280×1280像素）
<scene_id>/<frame_id:06d>.200-2.png
...
<scene_id>/<frame_id:06d>.200-12.png
<scene_id>/<frame_id:06d>.metadata.json # 姿态、采集装置校准信息与场景信息

每个样本的`metadata.json`包含以下字段：

| 字段 | 类型 | 描述 |
|-------|------|-------------|
| `scene_id` | 字符串 | 录制标识符 |
| `frame_id` | 整数 | 帧索引（从0开始计数） |
| `timestamp_ns` | 整数 | 纳秒级帧时间戳 |
| `pose` | 列表 | 4×4设备到世界坐标系变换矩阵（姿态不可用时该字段缺失） |
| `rig_calibration` | 对象 | 每相机内参（`K`）与外参（`E`） |
| `source` | 字符串 | 采集项目：`OVD_M1`（实验室场景，513段录制）、`OVD_M2`（公寓场景，414段）、`DD4`（29段） |
| `lux_bins` | 字符串 | 光照等级：`51-75`、`76-100`、`101-200`、`201-400`、`401-1000`、`1001+` |
| `tags` | 列表 | 场景多样性标签 |

## Parquet元数据架构

每一行代表单帧数据（所有12台相机同一时间戳的画面集合）：

| 列名 | 数据类型 | 描述 |
|--------|------|-------------|
| `scene_id` | 字符串 | 录制标识符 |
| `frame_id` | int32 | 单段录制内的帧索引（从0开始计数；每场景帧数范围为404-583） |
| `timestamp_ns` | int64 | 纳秒级帧时间戳 |
| `source` | 字符串 | 采集项目：`OVD_M1`（实验室场景）、`OVD_M2`（公寓场景）、`DD4` |
| `lux_bins` | 字符串 | 光照等级：`51-75`、`76-100`、`101-200`、`201-400`、`401-1000`、`1001+` |
| `tags` | 字符串 | JSON格式的场景多样性标签列表（共85个独特标签，涵盖服饰、家具、光照、姿态、物体等类别） |
| `shard_name` | 字符串 | 包含该帧图像的tar分片相对路径（例如：`shards/train/<scene_id>/<scene_id>-0002.tar`） |
| `pose` | 字符串 | JSON格式的4×4设备到世界坐标系变换矩阵（姿态不可用时为null） |
| `rig_calibration` | 字符串 | JSON格式的每相机内参（`K`：3×3矩阵）与外参（`E`：4×4矩阵），每个场景固定（每帧重复以方便使用） |

### 校准细节

`rig_calibration`列包含以相机名称（`200-1`至`200-12`）为键的JSON对象，每个键对应以下内容：
- **`K`**：3×3内参矩阵（校正针孔投影，水平视场角120°）
- **`E`**：4×4外参矩阵（相机到设备坐标系变换）

`pose`列包含4×4设备到世界坐标系变换矩阵，随头戴设备运动逐帧变化。

## 使用方法

`load_dataset`仅返回帧级元数据（姿态、校准信息、场景信息）。图像存储在WebDataset格式的tar分片中，需使用`webdataset`库进行流式读取。请参阅`quickstart.ipynb`文件获取完整可运行示例。

### WebDataset（推荐用于训练）

流式读取tar分片以实现高吞吐量的顺序访问——无需逐文件调用API。请参阅笔记本文件获取完整的`decode_sample`实现代码。若要封装为PyTorch DataLoader：

python
dataset = wds.WebDataset(shard_urls, nodesplitter=wds.split_by_node, shardshuffle=True).map(decode_sample)
loader = torch.utils.data.DataLoader(dataset, batch_size=4, num_workers=4)

for batch in loader:
images = batch["images"] # (B, N_cams, 3, 1280, 1280)
break

### Parquet元数据（支持随机访问）

Parquet文件仅包含帧级元数据（姿态、校准信息、场景信息），图像存储在tar分片中。使用`shard_name`列可定位包含指定帧图像的tar文件。

python
shard_url = f"https://huggingface.co/datasets/facebook/ego-1k/resolve/main/{example['shard_name']}"

## 引用格式

bibtex
@inproceedings{ego1k2026,
title={{Ego-1K}: A Large-Scale Multiview Video Dataset for Egocentric Vision},
author={Jae Yong Lee and Daniel Scharstein and Akash Bapat and Hao Hu and Andrew Fu and Haoru Zhao and Paul Sammut and Xiang Li and Stephen Jeapes and Anik Gupta and Lior David and Saketh Madhuvarasu and Jay Girish Joshi and Jason Wither},
booktitle={CVPR},
year={2026}
}

搜集汇总

背景与挑战

背景概述

Ego-1K是一个大规模多视角视频数据集，包含956个由12个同步摄像头捕获的短时自我中心视频，总计约491K帧和5.9M图像，数据量约18TB。该数据集专注于手部动作和物体交互场景，旨在推动神经3D视频合成和动态场景理解的研究，并提供了训练和测试划分以支持方法基准测试。