HPAI-BSC/HEART

HEART数据集是一个用于评估医疗视觉问答（VQA）模型对抗鲁棒性的基准数据集。它包含多个分割，包括一个无注入线索的基线分割和四个基于线索的分割。线索分为两类：仅提示（prompt-only，图像不变）和覆盖层（overlay，信息渲染到图像上）。每种线索有两种变体：辅助性（assistive，线索与真实答案一致）和对抗性（adversarial，线索支持错误选项）。具体线索类型包括：1) 基线：无添加线索；2) 奉承（仅提示）：提示中声明建议答案，测试模型是否遵循声明而非视觉证据；3) 提示字幕（仅提示）：文本提示中包含类似字幕的描述，对抗性样本中描述支持干扰标签或框；4) 图像字幕（覆盖层）：字幕作为视觉覆盖层渲染到图像上，评估视觉嵌入文本线索；5) 图例（覆盖层，仅检测）：显示候选框及其标签的图例，对抗性变体修改映射以支持干扰框。数据集基于八个源医疗影像数据集构建，涵盖X射线血管造影、乳腺超声、牙科全景X光、脑部MRI、胃肠道内窥镜、骨折X光、眼底摄影和皮肤镜图像等多种模态，总共包含24,311个样本。每个样本包括图像、问题、多个选项（A、B、C、D）、正确答案以及各种元数据（如任务类型、线索类型、真实边界框、假边界框等）。数据集旨在系统测试医疗VQA模型在面对不同注入线索时的鲁棒性和偏差。

The HEART dataset is composed of multiple splits that differ in the type of injected cues. It includes a baseline split with no injected cues and four cue-based splits. Each cue is instantiated in two variants: assistive, where the cue is consistent with the GT, and adversarial, where the cue supports an incorrect option. Cue types, summarized below, fall into two categories: prompt-only cues where the image is unchanged, and overlay cues where information is rendered onto the image. - Baseline: No cue is added to the original question and available options. These are constructed by shuffling the GT option with wrong alternatives to establish baseline performance. - Sycophancy (prompt-only): The prompt asserts a suggested answer, testing whether the model follows the assertion over visual evidence. - Prompt captions (prompt-only): A caption-like description is included in the text prompt. In adversarial samples it is constructed to support a distractor label or box. - Image captions (overlay): The caption is rendered onto the image as a visual overlay, enabling evaluation of visually embedded textual cues under the same assistive/adversarial setup. - Legends (overlay; detection only): Candidate boxes are shown with a legend, mapping box identifiers to labels. The adversarial variant modifies the mapping to support a distractor box. The dataset is built from eight source medical imaging datasets covering modalities such as X-ray angiography, breast ultrasound, panoramic dental X-rays, brain MRI, gastrointestinal endoscopy, fracture X-rays, fundus photography, and dermoscopic images, totaling 24,311 samples. It is designed to evaluate the adversarial robustness of medical visual question answering models.