Haxxsh/SHROOMCAP2025_DataSet

--- license: mit task_categories: - text-classification language: - en - es - fr - it - hi tags: - hallucination-detection - nlp - xlm-roberta - shroom-cap - multilingual - fact-checking pretty_name: SHROOM-CAP 2025 Unified Hallucination Detection Dataset size_categories: - 100K<n<1M --- # 🍄 SHROOM-CAP 2025 Unified Hallucination Detection Dataset <div align="center">     </div> ## 📖 Dataset Description This dataset was created for the **SHROOM-CAP 2025 Shared Task** on multilingual scientific hallucination detection. It combines and unifies multiple hallucination detection datasets into a single, balanced training corpus for fine-tuning XLM-RoBERTa-Large. ### Key Features - **124,821 total samples** - 172x larger than original SHROOM training data - **Perfectly balanced** - 50% correct, 50% hallucinated - **Multilingual** - Supports English, Spanish, French, Italian, and Hindi - **Pre-processed** - Ready for immediate model training - **Multiple formats** - Available in JSONL and CSV ## 📊 Dataset Statistics | Metric | Value | |--------|-------| | Total Samples | 124,821 | | Training Samples | 112,338 (90%) | | Validation Samples | 12,483 (10%) | | Correct (Label 0) | 62,257 (49.9%) | | Hallucinated (Label 1) | 62,564 (50.1%) | ### By Source | Source | Samples | Description | |--------|---------|-------------| | `hallucination_dataset_100k` | 100,000 | AI-generated synthetic hallucination data | | `libreeval` | 24,097 | Multilingual evaluation dataset | | `shroom_trainset_v2` | 724 | Official SHROOM competition data | ### By Language | Language | Code | Samples | |----------|------|---------| | English | en | 124,181 | | Spanish | es | 160 | | French | fr | 160 | | Hindi | hi | 160 | | Italian | it | 160 | ## 📁 Dataset Files ``` ├── unified_training_data.jsonl # All samples (85MB) ├── train.jsonl # Training split 90% (77MB) ├── val.jsonl # Validation split 10% (8.6MB) └── unified_training_data.csv # CSV format (67MB) ``` ## 🏷️ Data Format Each sample contains the following fields: ```json { "text": "Question: Who painted the Mona Lisa? Context: A famous Renaissance portrait. Answer: The Mona Lisa was painted by Leonardo da Vinci.", "label": 0, "source": "hallucination_dataset_100k", "language": "en", "metadata": { "id": "12345", "category": "art", "difficulty": "easy" } } ``` ### Field Descriptions | Field | Type | Description | |-------|------|-------------| | `text` | string | The input text containing question, context, and answer | | `label` | int | 0 = CORRECT, 1 = HALLUCINATED | | `source` | string | Original dataset source | | `language` | string | ISO language code (en, es, fr, it, hi) | | `metadata` | dict | Additional information (optional) | ## 🚀 Usage ### Loading with Hugging Face Datasets ```python from datasets import load_dataset # Load the full dataset dataset = load_dataset("Haxxsh/SHROOMCAP2025_DataSet") # Load specific splits train_data = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files="train.jsonl") val_data = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files="val.jsonl") ``` ### Loading with Pandas ```python import pandas as pd # Load CSV version df = pd.read_csv("hf://datasets/Haxxsh/SHROOMCAP2025_DataSet/unified_training_data.csv") # Load JSONL version df = pd.read_json("hf://datasets/Haxxsh/SHROOMCAP2025_DataSet/train.jsonl", lines=True) ``` ### Training Example ```python from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments from datasets import load_dataset # Load dataset dataset = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files={"train": "train.jsonl", "validation": "val.jsonl"}) # Load model and tokenizer model_name = "xlm-roberta-large" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2) # Tokenize def tokenize_function(examples): return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=256) tokenized_dataset = dataset.map(tokenize_function, batched=True) # Train training_args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=32, learning_rate=2e-5, evaluation_strategy="epoch", ) trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_dataset["train"], eval_dataset=tokenized_dataset["validation"], ) trainer.train() ``` ## 🏆 Competition Results This dataset was used to train our XLM-RoBERTa-Large model for the SHROOM-CAP 2025 competition, achieving: | Language | Rank | Factuality F1 | |----------|------|---------------| | Gujarati (zero-shot) | **2nd** 🥈 | 0.5107 | | Bengali | 4th | 0.4449 | | Hindi | 4th | 0.4906 | | Spanish | 5th | 0.4938 | | French | 5th | 0.4771 | | Telugu | 5th | 0.4738 | | Malayalam | 5th | 0.4704 | | English | 6th | 0.4246 | | Italian | 5th | 0.3149 | ## 📚 Citation If you use this dataset, please cite our paper: ```bibtex @inproceedings{rathva2025agi, title={AGI Team at SHROOM-CAP: Data-Centric Approach to Multilingual Hallucination Detection using XLM-RoBERTa}, author={Rathva, Harsh and Mishra, Pruthwik and Malviya, Shrikant}, booktitle={Proceedings of the SHROOM-CAP 2025 Shared Task}, year={2025}, institution={Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India} } ``` ## 🔗 Related Resources - **Model Weights:** [Haxxsh/XLMRHallucinationDetectorSHROOMCAP](https://huggingface.co/Haxxsh/XLMRHallucinationDetectorSHROOMCAP) - **Code Repository:** [github.com/ezylopx5/SHROOM-CAP2025](https://github.com/ezylopx5/SHROOM-CAP2025) - **SHROOM-CAP Task:** [SHROOM Shared Task](https://helsinki-nlp.github.io/shroom/) ## 📝 License This dataset is released under the MIT License. ## 👥 Authors - **Harsh Rathva** - SVNIT, Surat, India - **Pruthwik Mishra** - SVNIT, Surat, India - **Shrikant Malviya** - SVNIT, Surat, India --- <div align="center"> Made with ❤️ by the AGI Team at SVNIT </div>license: MIT许可证（MIT License） task_categories: - 文本分类（text-classification） language: - 英语（en） - 西班牙语（es） - 法语（fr） - 意大利语（it） - 印地语（hi） tags: - 幻觉检测（hallucination-detection） - 自然语言处理（Natural Language Processing，简称NLP） - XLM-RoBERTa（XLM-RoBERTa） - SHROOM-CAP（SHROOM-CAP） - 多语言（multilingual） - 事实核查（fact-checking） pretty_name: SHROOM-CAP 2025 统一幻觉检测数据集（SHROOM-CAP 2025 Unified Hallucination Detection Dataset） size_categories: - 100,000 < 样本数量 < 1,000,000 --- # 🍄 SHROOM-CAP 2025 统一幻觉检测数据集 <div align="center">     </div> ## 📖 数据集说明 本数据集专为**SHROOM-CAP 2025共享任务（SHROOM-CAP 2025 Shared Task）**（多语言科学幻觉检测）打造，将多个幻觉检测数据集整合统一为单个平衡的训练语料库，用于微调XLM-RoBERTa-Large。 ### 核心特性 - **总样本量124,821**——是原始SHROOM训练数据的172倍 - **完美标签平衡**——50%为正确样本，50%为幻觉样本 - **多语言支持**——覆盖英语、西班牙语、法语、意大利语及印地语 - **预处理就绪**——可直接用于模型训练 - **多格式兼容**——提供JSONL与CSV两种格式 ## 📊 数据集统计 | 指标 | 数值 | |--------|-------| | 总样本量 | 124,821 | | 训练样本 | 112,338（占比90%） | | 验证样本 | 12,483（占比10%） | | 正确样本（标签0） | 62,257（占比49.9%） | | 幻觉样本（标签1） | 62,564（占比50.1%） | ### 按来源划分 | 来源 | 样本量 | 说明 | |--------|---------|-------------| | `hallucination_dataset_100k` | 100,000 | AI生成的合成幻觉数据 | | `libreeval` | 24,097 | 多语言评估数据集 | | `shroom_trainset_v2` | 724 | 官方SHROOM竞赛数据集 | ### 按语言划分 | 语言 | 代码 | 样本量 | |----------|------|---------| | 英语 | en | 124,181 | | 西班牙语 | es | 160 | | 法语 | fr | 160 | | 印地语 | hi | 160 | | 意大利语 | it | 160 | ## 📁 数据集文件 ├── unified_training_data.jsonl # 全量样本文件（85MB） ├── train.jsonl # 训练集拆分（占90%，77MB） ├── val.jsonl # 验证集拆分（占10%，8.6MB） └── unified_training_data.csv # CSV格式数据集（67MB） ## 🏷️ 数据格式 每个样本包含以下字段： json { "text": "Question: Who painted the Mona Lisa? Context: A famous Renaissance portrait. Answer: The Mona Lisa was painted by Leonardo da Vinci.", "label": 0, "source": "hallucination_dataset_100k", "language": "en", "metadata": { "id": "12345", "category": "art", "difficulty": "easy" } } ### 字段说明 | 字段 | 类型 | 说明 | |-------|------|-------------| | `text` | 字符串 | 包含问题、上下文与答案的输入文本 | | `label` | 整数 | 0 = 正确样本，1 = 幻觉样本 | | `source` | 字符串 | 原始数据集来源 | | `language` | 字符串 | ISO语言代码（en、es、fr、it、hi） | | `metadata` | 字典 | 附加信息（可选） | ## 🚀 使用方法 ### 使用Hugging Face Datasets加载 python from datasets import load_dataset # 加载全量数据集 dataset = load_dataset("Haxxsh/SHROOMCAP2025_DataSet") # 加载指定拆分的数据集 train_data = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files="train.jsonl") val_data = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files="val.jsonl") ### 使用Pandas加载 python import pandas as pd # 加载CSV格式数据集 df = pd.read_csv("hf://datasets/Haxxsh/SHROOMCAP2025_DataSet/unified_training_data.csv") # 加载JSONL格式数据集 df = pd.read_json("hf://datasets/Haxxsh/SHROOMCAP2025_DataSet/train.jsonl", lines=True) ### 训练示例 python from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments from datasets import load_dataset # 加载数据集 dataset = load_dataset("Haxxsh/SHROOMCAP2025_DataSet", data_files={"train": "train.jsonl", "validation": "val.jsonl"}) # 加载模型与分词器 model_name = "xlm-roberta-large" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2) # 分词函数 def tokenize_function(examples): return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=256) tokenized_dataset = dataset.map(tokenize_function, batched=True) # 训练参数配置 training_args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=32, learning_rate=2e-5, evaluation_strategy="epoch", ) # 初始化训练器 trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_dataset["train"], eval_dataset=tokenized_dataset["validation"], ) # 启动训练 trainer.train() ## 🏆 竞赛结果 本数据集被用于训练我们的XLM-RoBERTa-Large模型参加SHROOM-CAP 2025竞赛，取得以下成绩： | 语言 | 排名 | 事实性F1值 | |----------|------|---------------| | 古吉拉特语（零样本） | **第2名** 🥈 | 0.5107 | | 孟加拉语 | 第4名 | 0.4449 | | 印地语 | 第4名 | 0.4906 | | 西班牙语 | 第5名 | 0.4938 | | 法语 | 第5名 | 0.4771 | | 泰卢固语 | 第5名 | 0.4738 | | 马拉雅拉姆语 | 第5名 | 0.4704 | | 英语 | 第6名 | 0.4246 | | 意大利语 | 第5名 | 0.3149 | ## 📚 引用 若您使用本数据集，请引用如下论文： bibtex @inproceedings{rathva2025agi, title={AGI Team at SHROOM-CAP: Data-Centric Approach to Multilingual Hallucination Detection using XLM-RoBERTa}, author={Rathva, Harsh and Mishra, Pruthwik and Malviya, Shrikant}, booktitle={Proceedings of the SHROOM-CAP 2025 Shared Task}, year={2025}, institution={Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India} } ## 🔗 相关资源 - **模型权重**：[Haxxsh/XLMRHallucinationDetectorSHROOMCAP](https://huggingface.co/Haxxsh/XLMRHallucinationDetectorSHROOMCAP) - **代码仓库**：[github.com/ezylopx5/SHROOM-CAP2025](https://github.com/ezylopx5/SHROOM-CAP2025) - **SHROOM-CAP任务**：[SHROOM共享任务](https://helsinki-nlp.github.io/shroom/) ## 📝 许可证 本数据集采用MIT许可证发布。 ## 👥 作者 - **Harsh Rathva**——印度苏拉特 Sardar Vallabhbhai国家技术学院（SVNIT） - **Pruthwik Mishra**——印度苏拉特 Sardar Vallabhbhai国家技术学院（SVNIT） - **Shrikant Malviya**——印度苏拉特 Sardar Vallabhbhai国家技术学院（SVNIT） --- <div align="center"> 由SVNIT的AGI（通用人工智能）团队用心制作 ❤️ </div>