InternVL-SA-1B-Caption

# Dataset Card for InternVL-SA-1B-Caption

## Overview
The InternVL-SA-1B-Caption Dataset is a bilingual dataset created using the [InternVL2-Llama3-76B](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) model. The dataset contains **12 million** image-caption pairs in both English and Chinese. All images are sourced from [Meta’s SA-1B dataset](https://ai.meta.com/datasets/segment-anything/), and captions were generated using specific prompts designed to minimize hallucinations and ensure accurate descriptions based on visible image content. The dataset is intended for use in tasks related to image-captioning, multilingual learning, and multimodal applications.

## Construction
The dataset was built by captioning images from the SA-1B dataset in both English and Chinese. We used the `InternVL2-Llama3-76B` model to generate accurate captions using the following prompts:

- **English Prompt**: "Generate an accurate, single-paragraph description based on the given image. Do not use multiple paragraphs or line breaks. Avoid generating speculative content. Ensure that the description is based on clearly visible information in the image and avoid any over-speculation."
- **Chinese Prompt**: "生成基于给定图像的准确单段描述。请勿使用多段或换行。避免生成带有推测性质的内容。确保描述基于图像中清晰可见的信息，避免过度推测。"

To speed up the generation process, we utilized the [lmdeploy](https://github.com/InternLM/lmdeploy) inference framework, achieving nearly a 10x acceleration compared to the standard Transformers pipeline. After caption generation, data was filtered to remove outliers, such as abnormally short or long captions and repeated samples.

## Image Source (SA-1B)
All images are sourced from [Meta’s SA-1B dataset](https://ai.meta.com/datasets/segment-anything/). To access the SA-1B dataset, please follow Meta’s official guidelines for dataset download and usage.

## Files in the Dataset
The dataset is divided into four primary files based on image count and language:

| File Name | Data Size | Type | Language |
|-----------------------------------------------------|-----------|--------------------------|----------|
| `internvl_sa1b_caption_11m_single_image_en.jsonl` | 11M | Single-image Caption | English |
| `internvl_sa1b_caption_1m_single_image_zh.jsonl` | 1M | Single-image Caption | Chinese |
| `internvl_sa1b_caption_77k_multi_image_en.jsonl` | 77K | Multi-image Caption | English |
| `internvl_sa1b_caption_77k_multi_image_zh.jsonl` | 77K | Multi-image Caption | Chinese |

- **Single-image captions**: These files contain captions where each caption corresponds to a single image.
- **Multi-image captions**: These files contain captions that describe multiple images.

## Citation
If you use this dataset in your research, please consider citing:

```
@article{chen2023internvl,
title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
journal={arXiv preprint arXiv:2312.14238},
year={2023}
}
@article{chen2024far,
title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
journal={arXiv preprint arXiv:2404.16821},
year={2024}
}
```

# InternVL-SA-1B-Caption 数据集卡片

## 概述
InternVL-SA-1B-Caption 数据集是一款双语数据集，依托[InternVL2-Llama3-76B](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B)模型构建而成。该数据集包含**1200万**组中英双语图像-文本对（image-caption pair），所有图像均源自[Meta的SA-1B数据集（Meta’s SA-1B dataset）](https://ai.meta.com/datasets/segment-anything/)，其描述文本通过定制化提示生成，旨在最大限度减少模型幻觉，确保生成的描述严格基于图像中可见的内容。本数据集可用于图像描述生成、多语言学习以及多模态应用相关任务。

## 数据集构建
本数据集通过对SA-1B数据集中的图像生成中英双语描述文本构建完成。我们使用`InternVL2-Llama3-76B`模型，结合以下提示词生成准确的图像描述：
- **英文提示词**："请基于给定图像生成准确的单段落描述，请勿使用多段或换行，避免生成推测性内容，确保描述仅基于图像中清晰可见的信息，严禁过度臆测。"
- **中文提示词**："生成基于给定图像的准确单段描述。请勿使用多段或换行。避免生成带有推测性质的内容。确保描述基于图像中清晰可见的信息，避免过度推测。"

为加速生成流程，我们采用了[lmdeploy](https://github.com/InternLM/lmdeploy)推理框架，相比标准Transformers推理流水线实现了近10倍的速度提升。图像描述生成完成后，我们对数据进行了过滤，移除了异常样本（如长度异常过短或过长的描述文本，以及重复样本）。

## 图像来源（SA-1B）
所有图像均源自[Meta的SA-1B数据集（Meta’s SA-1B dataset）](https://ai.meta.com/datasets/segment-anything/)。如需获取SA-1B数据集，请遵循Meta官方发布的数据集下载与使用指南。

## 数据集文件结构
本数据集根据图像数量与语言分为四个主要文件：

| 文件名 | 数据规模 | 数据类型 | 语言 |
|-----------------------------------------------------|-----------|--------------------------|----------|
| `internvl_sa1b_caption_11m_single_image_en.jsonl` | 11M | 单图像描述 | 英语 |
| `internvl_sa1b_caption_1m_single_image_zh.jsonl` | 1M | 单图像描述 | 中文 |
| `internvl_sa1b_caption_77k_multi_image_en.jsonl` | 77K | 多图像描述 | 英语 |
| `internvl_sa1b_caption_77k_multi_image_zh.jsonl` | 77K | 多图像描述 | 中文 |

- **单图像描述**：此类文件中的每条描述文本对应单张图像。
- **多图像描述**：此类文件中的描述文本用于描述多张图像。

## 引用
如果您在研究中使用本数据集，请引用以下文献：

@article{chen2023internvl,
title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
journal={arXiv preprint arXiv:2312.14238},
year={2023}
}
@article{chen2024far,
title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
journal={arXiv preprint arXiv:2404.16821},
year={2024}
}

搜集汇总

背景与挑战

背景概述

InternVL-SA-1B-Caption是一个由thomas创建的数据集，采用Apache License 2.0许可证，大小为12.52GB，于2024年12月9日更新，目前下载量为108次。