shivaniachary123/pii-masking-400k

Ai4Privacy PII 300k数据集是世界上最大的开放隐私掩码数据集，用于训练和评估模型从文本中移除个人可识别信息（PII）和敏感信息，特别适用于AI助手和大语言模型（LLMs）的上下文。数据集为合成数据，使用专有算法生成，无隐私侵犯风险。包含406,896个条目，总令牌数为20,564,179个，其中PII令牌为2,357,029个。支持6种语言：英语、意大利语、法语、德语、荷兰语和西班牙语，并覆盖8个司法管辖区：英国、美国、意大利、法国、瑞士、荷兰、德国和西班牙。数据集分为训练集（325,517条，80%）和验证集（81,379条，20%）。公共数据集包含17个PII类别，扩展数据集包含63个PII类别，提供更全面的敏感信息覆盖。数据格式为JSON对象，包括source_text（含PII的原始文本）、target_text（掩码后的文本）、privacy_mask（隐私掩码标签信息）、span_labels（PII在文本中的精确跨度映射）、mberttokens（多语言BERT令牌化结果）、mbert_bio_labels（BIO标签序列）、id（条目ID）、language（语言代码）、locale（区域代码）和split（数据集划分类型）。兼容多种机器学习任务，如令牌分类、文本生成、文本到文本生成等，适用于聊天机器人、客户支持系统、电子邮件过滤、数据匿名化、社交平台隐私保护等多种应用场景。数据集基于p5y隐私框架构建，遵循意识、保护和质量保证的三步流程。许可证为其他（详见license.md），学术使用鼓励并需适当引用，商业使用需联系授权。

The Ai4Privacy PII 300k Dataset is the worlds largest open dataset for privacy masking, designed to train and evaluate models for removing personally identifiable information (PII) and sensitive information from text, especially in the context of AI assistants and LLMs. It is synthetic data generated using proprietary algorithms with no privacy violations. The dataset contains 406,896 entries, with a total of 20,564,179 tokens and 2,357,029 PII tokens. It supports 6 languages: English, Italian, French, German, Dutch, and Spanish, and covers 8 jurisdictions: United Kingdom, United States, Italy, France, Switzerland, Netherlands, Germany, and Spain. The dataset is split into training set (325,517 entries, 80%) and validation set (81,379 entries, 20%). The public dataset includes 17 PII classes, while the extended dataset includes 63 PII classes for more comprehensive coverage. Data is formatted as JSON objects with fields such as source_text (original text with PII), target_text (masked version), privacy_mask (privacy mask labels), span_labels (exact PII spans), mberttokens (multilingual BERT tokenization), mbert_bio_labels (BIO labels), id (entry ID), language (language code), locale (locale code), and split (dataset split type). It is compatible with various machine learning tasks like token classification, text generation, and text-to-text generation, and applicable to use cases such as chatbots, customer support systems, email filtering, data anonymization, and social media privacy protection. The dataset is built on the p5y framework, following a three-step approach of awareness, protection, and quality assurance. The license is other (see license.md), with academic use encouraged with proper citation and commercial use requiring contact for licensing.