MMEB-eval
收藏资源简介：
大规模多模态嵌入基准（MMEB）数据集旨在评估多模态嵌入模型的能力。该基准涵盖4个元任务和36个精心挑选的评估数据集。每个数据集包含1000个用于评估的示例，每个示例包含一个查询和一组目标，查询和目标可以是图像和文本的组合。第一个目标是真实答案。数据集的许可证为MIT，语言为英语，标签为排名，大小类别为10K<n<100K。此外，数据集提供了统计信息和排行榜，以及提交和引用指南。
The Large-scale Multimodal Embedding Benchmark (MMEB) dataset is designed to evaluate the capabilities of multimodal embedding models. This benchmark encompasses 4 meta-tasks and 36 carefully selected evaluation datasets. Each dataset includes 1,000 evaluation samples, with each sample containing a query and a set of targets. Both the query and the targets can be combinations of images and text. The first target serves as the ground-truth answer. The dataset is licensed under MIT License, uses English as its language, adopts ranking as its annotation label, and falls into the size category of 10K < n < 100K. Additionally, the dataset provides statistical information, leaderboards, as well as submission and citation guidelines.
Massive Multimodal Embedding Benchmark (MMEB)
概述
MMEB 是一个大规模多模态嵌入基准数据集，包含 36 个数据集，用于评估多模态嵌入模型的能力。每个数据集包含 1000 个测试样本，每个样本包含一个查询和一组目标，查询和目标可以是图像和文本的任意组合。
数据集配置
数据集列表
- A-OKVQA
- CIFAR-100
- CIRR
- ChartQA
- Country211
- DocVQA
- EDIS
- FashionIQ
- GQA
- HatefulMemes
- ImageNet-1K
- ImageNet-A
- ImageNet-R
- InfographicsVQA
- MSCOCO
- MSCOCO_i2t
- MSCOCO_t2i
- N24News
- NIGHTS
- OK-VQA
- OVEN
- ObjectNet
- Place365
- RefCOCO
- RefCOCO-Matching
- SUN397
- ScienceQA
- TextVQA
- VOC2007
- VisDial
- Visual7W
- Visual7W-Pointing
- VisualNews_i2t
- VisualNews_t2i
- VizWiz
- WebQA
- Wiki-SS-NQ
数据集特征
每个数据集包含以下特征：
qry_text: 查询文本，类型为字符串。qry_img_path: 查询图像路径，类型为字符串。tgt_text: 目标文本序列，类型为字符串。tgt_img_path: 目标图像路径序列，类型为字符串。
数据集分割
每个数据集仅包含一个测试集分割，包含 1000 个样本。
数据集大小
每个数据集的测试集大小和下载大小如下：
| 数据集名称 | 测试集大小 (bytes) | 下载大小 (bytes) |
|---|---|---|
| A-OKVQA | 14048199 | 1168340 |
| CIFAR-100 | 1519890 | 20544 |
| CIRR | 70162098 | 1565489 |
| ChartQA | 14354641 | 1434448 |
| Country211 | 3678000 | 31556 |
| DocVQA | 23044459 | 1734476 |
| EDIS | 184208708 | 3350382 |
| FashionIQ | 71169665 | 1729457 |
| GQA | 40809641 | 1764457 |
| HatefulMemes | 184890 | 9972 |
| ImageNet-1K | 28773890 | 185019 |
| ImageNet-A | 28772890 | 147780 |
| ImageNet-R | 3456890 | 23656 |
| InfographicsVQA | 19114439 | 1439837 |
| MSCOCO | 97759085 | 1681753 |
| MSCOCO_i2t | 60201740 | 1785583 |
| MSCOCO_t2i | 87127008 | 1296167 |
| N24News | 630658 | 110698 |
| NIGHTS | 75116000 | 1528646 |
| OK-VQA | 15332578 | 1564823 |
| OVEN | 717934263 | 406792141 |
| ObjectNet | 2036000 | 27132 |
| Place365 | 7045000 | 89866 |
| RefCOCO | 96493941 | 1858145 |
| RefCOCO-Matching | 145712476 | 2879385 |
| SUN397 | 7990000 | 118447 |
| ScienceQA | 23870406 | 958782 |
| TextVQA | 17435986 | 1571656 |
| VOC2007 | 368000 | 13813 |
| VisDial | 67989850 | 1730820 |
| Visual7W | 22047066 | 1564788 |
| Visual7W-Pointing | 94906832 | 1299380 |
| VisualNews_i2t | 118329649 | 81491360 |
| VisualNews_t2i | 97176206 | 1763677 |
| VizWiz | 20550246 | 1425789 |
| WebQA | 197701404 | 3257136 |
| Wiki-SS-NQ | 74583207 | 1900579 |
许可证
MIT 许可证
语言
英语
标签
- 排名
数据集名称
MMEB
数据集大小分类
10K < n < 100K
