flax-community/conceptual-12m-multilingual-marian

--- language: - en - de - es - fr --- This dataset is created from subset of [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions/). The original dataset has 12M captions but this dataset has around 10M image, caption pairs in different languages with 2.5M unique images. This dataset has captions translated from English to Spanish, German, French using language specific English to [Marian](https://huggingface.co/Helsinki-NLP) models. Data distribution is following: `train_file_marian_final.tsv`: 10010625 captions (2502656 captions of English, German, Spanish, French each) <br /> `val_file_marian_final.tsv`: 110592 captions (27648 captions of English, German, Spanish, French each)

原始信息汇总

数据集概述

数据集来源

本数据集是从Conceptual Captions中提取的一个子集，原始数据集包含1200万条描述，而本数据集包含约1000万对图像和描述，涉及多种语言。

数据集内容

• 图像数量：250万张独特图像。
• 描述语言：英语、德语、西班牙语、法语。
• 描述翻译：所有非英语描述均由英语通过Marian模型翻译而来。

数据分布

• 训练集：train_file_marian_final.tsv，包含10010625条描述，每种语言各2502656条。
• 验证集：val_file_marian_final.tsv，包含110592条描述，每种语言各27648条。