sled-umich/ROPE
收藏资源简介：
ROPE数据集旨在通过利用现有的全景分割数据集（如MSCOCO-Panoptic和ADE20K）来评估和分析多目标幻觉现象。数据集包含多样对象及其实例级语义注释，分为训练和验证两部分，每部分根据图像中对象类别的分布进一步细分为同质、异质、野外和对抗性子集。文件结构包括图像和JSON文件，图像分为原始图像和带有边界框的可视化图像。JSON文件结构详细描述了图像文件夹、文件名、来源、尺寸、分割信息、对象信息等。
The ROPE dataset is designed to evaluate and analyze multi-object hallucination in large vision-language models (LVLMs) by leveraging existing panoptic segmentation datasets such as MSCOCO-Panoptic and ADE20K. It includes diverse objects and their instance-level semantic annotations. The dataset is divided into several subsets based on the distribution of object classes within each image at test time, including Homogeneous, Heterogeneous, In-the-Wild, and Adversarial subsets. The dataset is structured into training and validation directories, each containing images divided by their object class distributions. Each image directory includes visualizations of bounding boxes and raw images, further categorized into ADE and COCO sources. The dataset also includes JSON files detailing the structure and features of the dataset.
数据集概述
基本信息
- 任务类别: 问答、文本分类
- 许可协议: Apache 2.0
- 语言: 英语
- 数据量: 1K<n<10K
数据集构造
数据集基于MSCOCO-Panoptic和ADE20K构建，旨在评估和分析多对象幻觉现象。数据集分为以下几个子集：
- Homogeneous: 所有测试对象属于同一类别。
- Heterogeneous: 所有测试对象属于不同类别。
- In-the-Wild: 混合分布，测试对象随机选择和排序。
- Adversarial: 设计用于挑战模型，包含困难的对象分布。
数据统计
训练数据统计
| 数据集 | 总数 | COCO | ADE |
|---|---|---|---|
| Wild | 1539 | 732 | 807 |
| Hom. | 312 | 168 | 144 |
| Het. | 400 | 200 | 200 |
| Adv. | 168 | 54 | 114 |
验证数据统计
| 数据集 | 总数 | COCO | ADE |
|---|---|---|---|
| Wild | 1172 | 547 | 625 |
| Het. | 246 | 76 | 170 |
| Hom. | 490 | 289 | 201 |
| Adv. | 334 | 170 | 164 |
数据文件结构
数据集分为训练和验证目录，每个目录包含按对象类别分布划分的图像。每个图像目录包含边界框(bbox)和原始图像(raw)的可视化，进一步分为ADE和COCO来源。
arduino
ROPE/
│
├── train/
│ ├── image/
│ │ ├── AAAAB-images/
│ │ │ ├── bbox/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ │ ├── raw/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ ├── BAAAA-images/
│ │ │ ├── bbox/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ │ ├── raw/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ ├── heterogenous-images/
│ │ │ ├── bbox/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ │ ├── raw/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ ├── homogenous-images/
│ │ │ ├── bbox/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ │ ├── raw/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ ├── mixed-images/
│ │ │ ├── bbox/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ │ │ ├── raw/
│ │ │ │ ├── ADE/
│ │ │ │ ├── COCO/
│ ├── AAAAB_data.json
│ ├── BAAAA_data.json
│ ├── merged_heterogenous_data.json
│ ├── merged_homogenous_data.json
│ ├── merged_mixed_data.json
│
├── validation/ #similar to train part
│ ├── image/
│ │ ├── AAAAB-images/
│ │ ├── BAAAA-images/
│ │ ├── heterogenous-images/
│ │ ├── homogenous-images/
│ │ ├── mixed-images/
│ ├── AAAAB_data.json
│ ├── BAAAA_data.json
│ ├── merged_heterogenous_data.json
│ ├── merged_homogenous_data.json
│ ├── merged_mixed_data.json
│
├── .gitattributes
├── README.md
├── train.zip
├── validation.zip
JSON文件结构
JSON文件包含以下特征：
folder: 文件夹名称，字符串类型。filename: 文件名，字符串类型。source: 数据来源，结构类型，包含database、image_id、coco_id、flickr_id字段。size: 图像尺寸，结构类型，包含width、height、depth字段。segmented: 是否分割，整数类型。objects: 对象列表，列表类型，每个对象包含name、object_id、difficult、bndbox、area、bbox_number字段。relations: 关系列表，字符串类型。object_set: 对象集合，字符串类型。data_source: 数据来源，字符串类型。
json { "features": [ { "name": "folder", "dtype": "string" }, { "name": "filename", "dtype": "string" }, { "name": "source", "dtype": "struct", "fields": [ { "name": "database", "dtype": "string" }, { "name": "image_id", "dtype": "string" }, { "name": "coco_id", "dtype": "string" }, { "name": "flickr_id", "dtype": "string" } ] }, { "name": "size", "dtype": "struct", "fields": [ { "name": "width", "dtype": "int32" }, { "name": "height", "dtype": "int32" }, { "name": "depth", "dtype": "int32" } ] }, { "name": "segmented", "dtype": "int32" }, { "name": "objects", "dtype": "list", "item": { "dtype": "struct", "fields": [ { "name": "name", "dtype": "string" }, { "name": "object_id", "dtype": "string" }, { "name": "difficult", "dtype": "int32" }, { "name": "bndbox", "dtype": "struct", "fields": [ { "name": "xmin", "dtype": "int32" }, { "name": "ymin", "dtype": "int32" }, { "name": "xmax", "dtype": "int32" }, { "name": "ymax", "dtype": "int32" } ] }, { "name": "area", "dtype": "int32" }, { "name": "bbox_number", "dtype": "int32" } ] } }, { "name": "relations", "dtype": "list", "item": { "dtype": "string" } }, { "name": "object_set", "dtype": "list", "item": { "dtype": "string" } }, { "name": "data_source", "dtype": "string" } ] }
