MCIF
收藏资源简介：
<p align="center"> <img src="./mcif_logo.png" width="600"> </p> ### Dataset Description, Collection, and Source MCIF (Multimodal Crosslingual Instruction Following) is a multilingual human-annotated benchmark based on scientific talks that is designed to evaluate instruction-following in crosslingual, multimodal settings over both short- and long-form inputs. MCIF spans three core modalities -- speech, vision, and text -- and four diverse languages (English, German, Italian, and Chinese), enabling a comprehensive evaluation of MLLMs' abilities to interpret instructions across languages and combine them with multimodal contextual information. ### License - CC-BY-4.0 ### Dataset Sources - **Repository:** [MCIF](https://github.com/hlt-mt/mcif) - **Paper:** [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://arxiv.org/abs/2507.19634) ## Dataset Structure ### Data Config This dataset contains **4 splits** organized by three dimensions following the split naming convention `{track}_{prompt_type}`. Track - Input duration: * `long`: Full-length, unsegmented inputs * `short`: Pre-segmented inputs Prompt Type - Prompt variation: * `fixed`: Standardized prompts across all examples * `mixed`: Includes prompt variations Please note that all splits share the same set of original input audio and video files. The splits are meant to facilitate testing various use cases. If you are interested in a specific task among those covered (Recognition, Translation, Question Answering, and Summarization), you can identify the relevant samples using the `task` field (respectively, `ASR`, `TRANS`, `QA`, and `SUM`) in each `sample` available from the [references](#references). The corresponding `id` field can then be used to match samples with their counterpart in this HuggingFace dataset. ### Dataset Fields | **Field** | **Type** | **Description** | |-----------------|------------|-----------------------------------------------| | `id` | `string` | Unique identifier for the sample. | | `audio` | `str` | In the `long` track: path to full talk-level audio. In the `short` track: path to pre-segmented audio. | | `video` | `str` | In the `long` track: path to full talk-level video. In the `short` track: path to pre-segmented video. | | `text` | `string` | Transcript of input. Only present in the `long` track. | | `prompt_{en, de, it, zh}` | `string` | Instruction in English, German, Italian, or Chinese. | | `metadata` | `string` | Meta data for question answering samples, in the format {qa_type={`A` (audio), `V` (visual), `AV` (audio-visual), `NA` (not answerable)} qa_origin={`Transcript`, `Abstract`, `General`}} | The audio/video paths are relative within this repo. You can download the data by cloning this repo: ``` git clone https://huggingface.co/datasets/FBK-MT/MCIF ``` ### References The references are available in `MCIF.{short,long}.{en,de,it,zh}.ref.xml.gz` (navigate to "Files and versions" tab or clone this repo). ### IWSLT 2025 Version Part of MCIF was used in the [IWSLT 2025 instruction-following track](https://iwslt.org/2025/instruction-following). This test data is available under branch `IWSLT2025`. You can access it by ``` dataset = load_dataset("FBK-MT/MCIF", "{en,de,it,zh}_{long,short}", revision="IWSLT2025") ``` ## Evaluation Please use the official evaluation scripts from the [MCIF GitHub Repo](https://github.com/hlt-mt/mcif). The references are also available there. ## Changelog ### Version 1.2 - Fixed summarization references ### Version 1.1 - Fixed German summarization prompt - Renamed files not to include version name in the filename ## Citation ``` @inproceedings{papi2026mcif, title={{MCIF}: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks}, author={Sara Papi and Maike Z{\"u}fle and Marco Gaido and Beatrice Savoldi and Danni Liu and Ioannis Douros and Luisa Bentivogli and Jan Niehues}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=PtPYZYfa0h} } ``` ## Dataset Card Contact [@spapi](https://huggingface.co/spapi) and [@danniliu](https://huggingface.co/danniliu)
<p align="center"><img src="./mcif_logo.png" width="600"></p> ### 数据集描述、收集与来源 MCIF（Multimodal Crosslingual Instruction Following，多模态跨语言指令遵循）是一个基于学术演讲的多语言人工标注基准测试集，旨在评估大语言模型（LLM）在跨语言、多模态场景下对长短格式输入的指令遵循能力。MCIF涵盖三大核心模态——语音、视觉与文本，以及四种覆盖广泛的语言（英语、德语、意大利语与中文），能够全面评估多模态大语言模型（Multimodal Large Language Model，MLLM）跨语言理解指令，并结合多模态上下文信息的能力。 ### 许可证 - CC-BY-4.0 ### 数据集来源 - **代码仓库**：[MCIF](https://github.com/hlt-mt/mcif) - **相关论文**：[MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://arxiv.org/abs/2507.19634) ## 数据集结构 ### 数据配置 本数据集包含**4个拆分集**，按照三个维度进行组织，拆分命名遵循`{track}_{prompt_type}`的格式约定。 - 赛道（Track）：输入时长维度 * `long`：完整未分段的输入内容 * `short`：预先分段的输入内容 - 提示类型（Prompt Type）：提示变体类型 * `fixed`：所有样本统一使用标准化提示词 * `mixed`：包含多种变体的提示词 请注意，所有拆分集共享同一套原始输入音频与视频文件，设置不同拆分集旨在便于测试多样化的应用场景。 ### 数据集字段 | **字段名** | **数据类型** | **字段说明** | |-----------------|------------|-----------------------------------------------| | `id` | `string` | 样本唯一标识符，前缀以`QA`（问答）、`SUM`（摘要生成）、`ASR`（语音转录）或`TRANS`（翻译）开头。 | | `audio` | `str` | 对于`long`赛道：指向完整演讲级音频文件的路径；对于`short`赛道：指向预先分段音频文件的路径。 | | `video` | `str` | 对于`long`赛道：指向完整演讲级视频文件的路径；对于`short`赛道：指向预先分段视频文件的路径。 | | `text` | `string` | 输入内容的转录文本，仅在`long`赛道中存在。 | | `prompt_{en, de, it, zh}` | `string` | 分别为英语、德语、意大利语或中文的指令提示词。 | | `metadata` | `string` | 问答样本的元数据，格式为`{qa_type={`A`（音频）, `V`（视觉）, `AV`（音视频）, `NA`（无法回答）}, qa_origin={`Transcript`, `Abstract`, `General`}}` | 音频与视频路径均为本仓库内的相对路径。 你可以通过克隆本仓库下载数据： git clone https://huggingface.co/datasets/FBK-MT/MCIF ### 参考数据集 参考数据集存储于`MCIF.{short,long}.{en,de,it,zh}.ref.xml.gz`（可前往仓库的"Files and versions"标签页或克隆本仓库获取）。 ### IWSLT 2025 版本 MCIF的部分内容被应用于[IWSLT 2025指令遵循赛道](https://iwslt.org/2025/instruction-following)。 该测试数据位于`IWSLT2025`分支下，你可以通过以下代码访问该数据集： dataset = load_dataset("FBK-MT/MCIF", "{en,de,it,zh}_{long,short}", revision="IWSLT2025") ## 评估 请使用[MCIF GitHub仓库](https://github.com/hlt-mt/mcif)提供的官方评估脚本，参考数据集也可在该仓库中获取。 ## 变更日志 ### 版本 1.1 - 修复了德语摘要生成提示词 - 重命名文件，使其文件名中不再包含版本号 ## 引用 @misc{papi2025mcifmultimodalcrosslingualinstructionfollowing, title={MCIF: 基于学术演讲的多模态跨语言指令遵循基准测试集}, author={Sara Papi and Maike Züfle and Marco Gaido and Beatrice Savoldi and Danni Liu and Ioannis Douros and Luisa Bentivogli and Jan Niehues}, year={2025}, eprint={2507.19634}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.19634}, } ## 数据集卡片联系人 [@spapi](https://huggingface.co/spapi) 与 [@danniliu](https://huggingface.co/danniliu)
