ai4privacy/pii-masking-300k

---
license: other
license_name: license.md
language:
- en
- fr
- de
- it
- es
- nl
task_categories:
- text-classification
- token-classification
- table-question-answering
- question-answering
- zero-shot-classification
- summarization
- feature-extraction
- text-generation
- text2text-generation
- translation
- fill-mask
- tabular-classification
- tabular-to-text
- table-to-text
- text-retrieval
- other
multilinguality:
- multilingual
tags:
- legal
- business
- psychology
- privacy
- gdpr
- euaiact
- aiact
- pii
- sensitive
size_categories:
- 100K<n<1M
pretty_name: Ai4Privacy PII 300k Dataset
source_datasets:
- original
configs:
- config_name: default
data_files:
- split: train
path: "data/train/*.jsonl"
- split: validation
path: "data/validation/*.jsonl"
---

# Purpose and Features

🌍 World's largest open dataset for privacy masking 🌎

The dataset is useful to train and evaluate models to remove personally identifiable and sensitive information from text, especially in the context of AI assistants and LLMs.

Key facts:
- OpenPII-220k text entries have **27 PII classes** (types of sensitive data), targeting **749 discussion subjects / use cases** split across education, health, and psychology. FinPII contains an additional **~20 types** tailored to insurance and finance. Kindly connect via [e-mail] for more information.
- Size: 30.4m text tokens in ~220'000+ examples with 7.6m PII tokens
- 2 more languages have been added! 6 languages in total with strong localisation in 8 jurisdictions.
- English (UK 🇬🇧 and USA 🇺🇸)
- French (France 🇫🇷 and Switzerland 🇨🇭)
- German (Germany 🇩🇪 and Switzerland 🇨🇭)
- Italian (Italy 🇮🇹 and Switzerland 🇨🇭)
- Dutch (Netherlands 🇳🇱)
- Spanish (Spain 🇪🇸)
- Introduced a training / validation split of: 79% - 21%
- Synthetic data generated using proprietary algorithms
- No privacy violations!
- Human-in-the-loop validated high quality dataset with transparent QA results (see [openpii_220k_27032024_QA.json](openpii_220k_27032024_QA.json)) with an ~98.3% token label accuracy of a random sample of 216 entries.
- PII-masking-300k is split into 2 sub-datasets: OpenPII-220k and FinPII-80k. The FinPII includes additional classes specific to Finance and Insurance. Please feel free to reach out to [e-mail] for additional details.

# Getting started

Option 1: Python
```terminal
pip install datasets
```
```python
from datasets import load_dataset
dataset = load_dataset("ai4privacy/pii-masking-300k")
```

# Text entry lengths and PII distributions
This is the 4th iteration of the pii-masking series datasets and we have further improved it by improving the average text entry length.

The current distribution of sensitive data and PII tokens:

# Compatible Machine Learning Tasks:
- Tokenclassification. Check out a HuggingFace's [guide on token classification](https://huggingface.co/docs/transformers/tasks/token_classification).
- [ALBERT](https://huggingface.co/docs/transformers/model_doc/albert), [BERT](https://huggingface.co/docs/transformers/model_doc/bert), [BigBird](https://huggingface.co/docs/transformers/model_doc/big_bird), [BioGpt](https://huggingface.co/docs/transformers/model_doc/biogpt), [BLOOM](https://huggingface.co/docs/transformers/model_doc/bloom), [BROS](https://huggingface.co/docs/transformers/model_doc/bros), [CamemBERT](https://huggingface.co/docs/transformers/model_doc/camembert), [CANINE](https://huggingface.co/docs/transformers/model_doc/canine), [ConvBERT](https://huggingface.co/docs/transformers/model_doc/convbert), [Data2VecText](https://huggingface.co/docs/transformers/model_doc/data2vec-text), [DeBERTa](https://huggingface.co/docs/transformers/model_doc/deberta), [DeBERTa-v2](https://huggingface.co/docs/transformers/model_doc/deberta-v2), [DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert), [ELECTRA](https://huggingface.co/docs/transformers/model_doc/electra), [ERNIE](https://huggingface.co/docs/transformers/model_doc/ernie), [ErnieM](https://huggingface.co/docs/transformers/model_doc/ernie_m), [ESM](https://huggingface.co/docs/transformers/model_doc/esm), [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon), [FlauBERT](https://huggingface.co/docs/transformers/model_doc/flaubert), [FNet](https://huggingface.co/docs/transformers/model_doc/fnet), [Funnel Transformer](https://huggingface.co/docs/transformers/model_doc/funnel), [GPT-Sw3](https://huggingface.co/docs/transformers/model_doc/gpt-sw3), [OpenAI GPT-2](https://huggingface.co/docs/transformers/model_doc/gpt2), [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode), [GPT Neo](https://huggingface.co/docs/transformers/model_doc/gpt_neo), [GPT NeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox), [I-BERT](https://huggingface.co/docs/transformers/model_doc/ibert), [LayoutLM](https://huggingface.co/docs/transformers/model_doc/layoutlm), [LayoutLMv2](https://huggingface.co/docs/transformers/model_doc/layoutlmv2), [LayoutLMv3](https://huggingface.co/docs/transformers/model_doc/layoutlmv3), [LiLT](https://huggingface.co/docs/transformers/model_doc/lilt), [Longformer](https://huggingface.co/docs/transformers/model_doc/longformer), [LUKE](https://huggingface.co/docs/transformers/model_doc/luke), [MarkupLM](https://huggingface.co/docs/transformers/model_doc/markuplm), [MEGA](https://huggingface.co/docs/transformers/model_doc/mega), [Megatron-BERT](https://huggingface.co/docs/transformers/model_doc/megatron-bert), [MobileBERT](https://huggingface.co/docs/transformers/model_doc/mobilebert), [MPNet](https://huggingface.co/docs/transformers/model_doc/mpnet), [MPT](https://huggingface.co/docs/transformers/model_doc/mpt), [MRA](https://huggingface.co/docs/transformers/model_doc/mra), [Nezha](https://huggingface.co/docs/transformers/model_doc/nezha), [Nyströmformer](https://huggingface.co/docs/transformers/model_doc/nystromformer), [QDQBert](https://huggingface.co/docs/transformers/model_doc/qdqbert), [RemBERT](https://huggingface.co/docs/transformers/model_doc/rembert), [RoBERTa](https://huggingface.co/docs/transformers/model_doc/roberta), [RoBERTa-PreLayerNorm](https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm), [RoCBert](https://huggingface.co/docs/transformers/model_doc/roc_bert), [RoFormer](https://huggingface.co/docs/transformers/model_doc/roformer), [SqueezeBERT](https://huggingface.co/docs/transformers/model_doc/squeezebert), [XLM](https://huggingface.co/docs/transformers/model_doc/xlm), [XLM-RoBERTa](https://huggingface.co/docs/transformers/model_doc/xlm-roberta), [XLM-RoBERTa-XL](https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl), [XLNet](https://huggingface.co/docs/transformers/model_doc/xlnet), [X-MOD](https://huggingface.co/docs/transformers/model_doc/xmod), [YOSO](https://huggingface.co/docs/transformers/model_doc/yoso)
- Text Generation: Mapping the unmasked_text to to the masked_text or privacy_mask attributes. Check out HuggingFace's [guide to fine-tunning](https://huggingface.co/docs/transformers/v4.15.0/training)
- [T5 Family](https://huggingface.co/docs/transformers/model_doc/t5), [Llama2](https://huggingface.co/docs/transformers/main/model_doc/llama2)

# Information regarding the rows:
- Each row represents a json object with a natural language text that includes placeholders for PII.
- Sample row:
- "source_text" shows a natural text containing PII
- "Subject: Group Messaging for Admissions Process\n\nGood morning, everyone,\n\nI hope this message finds you well. As we continue our admissions processes, I would like to update you on the latest developments and key information. Please find below the timeline for our upcoming meetings:\n\n- wynqvrh053 - Meeting at 10:20am\n- luka.burg - Meeting at 21\n- qahil.wittauer - Meeting at quarter past 13\n- gholamhossein.ruschke - Meeting at 9:47 PM\n- pdmjrsyoz1460 "
- "target_text" contains a masked version of the source text
- "Subject: Group Messaging for Admissions Process\n\nGood morning, everyone,\n\nI hope this message finds you well. As we continue our admissions processes, I would like to update you on the latest developments and key information. Please find below the timeline for our upcoming meetings:\n\n- [USERNAME] - Meeting at [TIME]\n- [USERNAME] - Meeting at [TIME]\n- [USERNAME] - Meeting at [TIME]\n- [USERNAME] - Meeting at [TIME]\n- [USERNAME] "
- "privacy_mask" contains the information explicit format for privacy mask labels
- [{"value": "wynqvrh053", "start": 287, "end": 297, "label": "USERNAME"}, {"value": "10:20am", "start": 311, "end": 318, "label": "TIME"}, {"value": "luka.burg", "start": 321, "end": 330, "label": "USERNAME"}, {"value": "21", "start": 344, "end": 346, "label": "TIME"}, {"value": "qahil.wittauer", "start": 349, "end": 363, "label": "USERNAME"}, {"value": "quarter past 13", "start": 377, "end": 392, "label": "TIME"}, {"value": "gholamhossein.ruschke", "start": 395, "end": 416, "label": "USERNAME"}, {"value": "9:47 PM", "start": 430, "end": 437, "label": "TIME"}, {"value": "pdmjrsyoz1460", "start": 440, "end": 453, "label": "USERNAME"}],
- "span_labels" displays the exact mapping spans of the private information within the text
- [[440, 453, "USERNAME"], [430, 437, "TIME"], [395, 416, "USERNAME"], [377, 392, "TIME"], [349, 363, "USERNAME"], [344, 346, "TIME"], [321, 330, "USERNAME"], [311, 318, "TIME"], [287, 297, "USERNAME"]],
- "mberttokens" indicates the breakdown of the text into tokens associated with multi-lingual bert
- ["Sub", "##ject", ":", "Group", "Mess", "##aging", "for", "Ad", "##mission", "##s", "Process", "Good", "morning", ",", "everyone", ",", "I", "hope", "this", "message", "finds", "you", "well", ".", "As", "we", "continue", "our", "admission", "##s", "processes", ",", "I", "would", "like", "to", "update", "you", "on", "the", "latest", "developments", "and", "key", "information", ".", "Please", "find", "below", "the", "time", "##line", "for", "our", "upcoming", "meetings", ":", "-", "w", "##yn", "##q", "##vr", "##h", "##0", "##53", "-", "Meeting", "at", "10", ":", "20", "##am", "-", "luka", ".", "bu", "##rg", "-", "Meeting", "at", "21", "-", "q", "##ahi", "##l", ".", "wit", "##tau", "##er", "-", "Meeting", "at", "quarter", "past", "13", "-", "gh", "##ola", "##mh", "##osse", "##in", ".", "rus", "##ch", "##ke", "-", "Meeting", "at", "9", ":", "47", "PM", "-", "p", "##d", "##m", "##jr", "##sy", "##oz", "##14", "##60"]
- mbert_bio_labels demonstrates the labels associated with the BIO labelling task in Machine Learning using the mbert tokens.
- ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "B-USERNAME", "I-USERNAME", "I-USERNAME", "O", "O", "O", "O", "O", "O", "O", "B-TIME", "I-TIME", "I-TIME", "O", "B-USERNAME", "I-USERNAME", "O", "O", "O", "B-TIME", "I-TIME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "O", "O", "O", "O", "B-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "O", "B-USERNAME", "I-USERNAME"],"
- "id": indicates the ID of the entry for future reference and feedback
- "40767A"
- "language": content of the language
- "English"
- "set": type of the machine learning set
- "train"

*note for the nested objects, we store them as string to maximise compability between various software.

# About Us:

At Ai4Privacy, we are commited to building the global seatbelt of the 21st century for Artificial Intelligence to help fight against potential risks of personal information being integrated into data pipelines.

Newsletter & updates: [www.Ai4Privacy.com](www.Ai4Privacy.com)
- Looking for ML engineers, developers, beta-testers, human in the loop validators (all languages)
- Integrations with already existing open solutions
- Ask us a question on discord: [https://discord.gg/kxSbJrUQZF](https://discord.gg/kxSbJrUQZF)

# Roadmap and Future Development

- Carbon neutral
- Additional benchmarking methods for NER
- Better multilingual and especially localisation
- Continuously increase the training and testing sets

# Known Issues

- Labelling error arise and are primarly around very coarse information types such as country, time, and titles. For example, country of Switzerland, the title of Dr. might not be labelled properly occasionally. We aim to reduce these mislabellings in further updates.

# Use Cases and Applications

**Chatbots**: Incorporating a PII masking model into chatbot systems can ensure the privacy and security of user conversations by automatically redacting sensitive information such as names, addresses, phone numbers, and email addresses.

**Customer Support Systems**: When interacting with customers through support tickets or live chats, masking PII can help protect sensitive customer data, enabling support agents to handle inquiries without the risk of exposing personal information.

**Email Filtering**: Email providers can utilize a PII masking model to automatically detect and redact PII from incoming and outgoing emails, reducing the chances of accidental disclosure of sensitive information.

**Data Anonymization**: Organizations dealing with large datasets containing PII, such as medical or financial records, can leverage a PII masking model to anonymize the data before sharing it for research, analysis, or collaboration purposes.

**Social Media Platforms**: Integrating PII masking capabilities into social media platforms can help users protect their personal information from unauthorized access, ensuring a safer online environment.

**Content Moderation**: PII masking can assist content moderation systems in automatically detecting and blurring or redacting sensitive information in user-generated content, preventing the accidental sharing of personal details.

**Online Forms**: Web applications that collect user data through online forms, such as registration forms or surveys, can employ a PII masking model to anonymize or mask the collected information in real-time, enhancing privacy and data protection.

**Collaborative Document Editing**: Collaboration platforms and document editing tools can use a PII masking model to automatically mask or redact sensitive information when multiple users are working on shared documents.

**Research and Data Sharing**: Researchers and institutions can leverage a PII masking model to ensure privacy and confidentiality when sharing datasets for collaboration, analysis, or publication purposes, reducing the risk of data breaches or identity theft.

**Content Generation**: Content generation systems, such as article generators or language models, can benefit from PII masking to automatically mask or generate fictional PII when creating sample texts or examples, safeguarding the privacy of individuals.

(...and whatever else your creative mind can think of)

# Licensing

Academic use is encouraged with proper citation provided it follows similar license terms*. Commercial entities should contact us at [e-mail] for licensing inquiries and additional data access.*

* Terms apply. See [LICENSE.md](LICENSE.md) for full details.

# Support and Maintenance

AI4Privacy is a project affiliated with [Ai Suisse SA](https://www.aisuisse.com/).

license: 其他
license_name: license.md
language:
- 英语（en）
- 法语（fr）
- 德语（de）
- 意大利语（it）
- 西班牙语（es）
- 荷兰语（nl）
task_categories:
- 文本分类
- Token分类
- 表格问答
- 问答
- 零样本分类
- 摘要
- 特征提取
- 文本生成
- 文本到文本生成
- 翻译
- 掩码填充
- 表格分类
- 表格到文本
- 表格转文本
- 文本检索
- 其他
multilinguality:
- 多语言
tags:
- 法律
- 商业
- 心理学
- 隐私
- 通用数据保护条例（GDPR）
- 欧盟人工智能法案（EU AI Act）
- 人工智能法案（AI Act）
- 个人可识别信息（PII）
- 敏感信息
size_categories:
- 100K<n<1M
pretty_name: Ai4Privacy PII 300k数据集
source_datasets:
- 原始数据集
configs:
- config_name: default
data_files:
- split: 训练集
path: "data/train/*.jsonl"
- split: 验证集
path: "data/validation/*.jsonl"

---

# 一、用途与特性

🌍 全球最大的开源个人可识别信息掩码数据集🌍

本数据集可用于训练和评估模型，以从文本中移除个人可识别信息（Personally Identifiable Information，简称PII）与敏感信息，尤其适用于AI智能体与大语言模型（LLM）场景。

核心要点：
- OpenPII-220k包含22万条文本条目，涵盖27类PII（敏感数据类型），覆盖教育、医疗与心理学领域的749个讨论主题/应用场景。FinPII则额外包含约20类专为保险与金融领域定制的敏感数据类型。如需更多信息，请通过[e-mail]联系我们。
- 规模：约22万条以上样本，总计3040万文本Token，其中PII Token达760万。
- 新增2种语言！本次数据集共支持6种语言，并在8个司法辖区实现了良好的本地化适配：
- 英语（英国🇬🇧与美国🇺🇸）
- 法语（法国🇫🇷与瑞士🇨🇭）
- 德语（德国🇩🇪与瑞士🇨🇭）
- 意大利语（意大利🇮🇹与瑞士🇨🇭）
- 荷兰语（荷兰🇳🇱）
- 西班牙语（西班牙🇪🇸）
- 训练集与验证集划分比例为79%：21%
- 采用专有算法生成合成数据
- 无隐私违规风险！
- 本数据集经过人机协同验证，质量可靠，公开了质检结果（详见[openpii_220k_27032024_QA.json](openpii_220k_27032024_QA.json)）：对216条随机样本的标注准确率约为98.3%。
- PII-masking-300k分为两个子数据集：OpenPII-220k与FinPII-80k。其中FinPII包含针对金融与保险领域的专属数据类别。如需更多细节，欢迎联系[e-mail]。

# 二、快速入门

方式一：Python环境
terminal
pip install datasets

python
from datasets import load_dataset
dataset = load_dataset("ai4privacy/pii-masking-300k")

# 三、文本条目长度与PII分布

本数据集是PII掩码系列数据集的第4个版本，我们通过优化平均文本条目长度进一步提升了数据集质量。

当前敏感数据与PII Token的分布情况：

# 四、兼容的机器学习任务

1. Token分类：可参考HuggingFace官方[Token分类任务指南](https://huggingface.co/docs/transformers/tasks/token_classification)。
支持的模型包括：ALBERT、BERT、BigBird、BioGpt、BLOOM、BROS、CamemBERT、CANINE、ConvBERT、Data2VecText、DeBERTa、DeBERTa-v2、DistilBERT、ELECTRA、ERNIE、ErnieM、ESM、Falcon、FlauBERT、FNet、Funnel Transformer、GPT-Sw3、OpenAI GPT-2、GPTBigCode、GPT Neo、GPT NeoX、I-BERT、LayoutLM、LayoutLMv2、LayoutLMv3、LiLT、Longformer、LUKE、MarkupLM、MEGA、Megatron-BERT、MobileBERT、MPNet、MPT、MRA、Nezha、Nyströmformer、QDQBert、RemBERT、RoBERTa、RoBERTa-PreLayerNorm、RoCBert、RoFormer、SqueezeBERT、XLM、XLM-RoBERTa、XLM-RoBERTa-XL、XLNet、X-MOD、YOSO

2. 文本生成：可实现未掩码文本到掩码文本或隐私掩码属性的映射。可参考HuggingFace官方[微调指南](https://huggingface.co/docs/transformers/v4.15.0/training)
支持的模型包括：T5系列、Llama2

# 五、数据行说明

每条数据行均为一个JSON对象，包含一段自然语言文本，其中包含PII占位符。

数据行示例：
- "source_text"：包含PII的原始自然文本
示例内容：
"Subject: Group Messaging for Admissions Process

Good morning, everyone,

I hope this message finds you well. As we continue our admissions processes, I would like to update you on the latest developments and key information. Please find below the timeline for our upcoming meetings:

- wynqvrh053 - Meeting at 10:20am
- luka.burg - Meeting at 21
- qahil.wittauer - Meeting at quarter past 13
- gholamhossein.ruschke - Meeting at 9:47 PM
- pdmjrsyoz1460 "
- "target_text"：原始文本的掩码版本
示例内容：
"Subject: Group Messaging for Admissions Process

Good morning, everyone,

I hope this message finds you well. As we continue our admissions processes, I would like to update you on the latest developments and key information. Please find below the timeline for our upcoming meetings:

- [USERNAME] - Meeting at [TIME]
- [USERNAME] - Meeting at [TIME]
- [USERNAME] - Meeting at [TIME]
- [USERNAME] - Meeting at [TIME]
- [USERNAME] "
- "privacy_mask"：隐私掩码标签的显式格式信息
示例内容：
[{"value": "wynqvrh053", "start": 287, "end": 297, "label": "USERNAME"}, {"value": "10:20am", "start": 311, "end": 318, "label": "TIME"}, {"value": "luka.burg", "start": 321, "end": 330, "label": "USERNAME"}, {"value": "21", "start": 344, "end": 346, "label": "TIME"}, {"value": "qahil.wittauer", "start": 349, "end": 363, "label": "USERNAME"}, {"value": "quarter past 13", "start": 377, "end": 392, "label": "TIME"}, {"value": "gholamhossein.ruschke", "start": 395, "end": 416, "label": "USERNAME"}, {"value": "9:47 PM", "start": 430, "end": 437, "label": "TIME"}, {"value": "pdmjrsyoz1460", "start": 440, "end": 453, "label": "USERNAME"}],
- "span_labels"：文本中私有信息的精确映射区间
示例内容：
[[440, 453, "USERNAME"], [430, 437, "TIME"], [395, 416, "USERNAME"], [377, 392, "TIME"], [349, 363, "USERNAME"], [344, 346, "TIME"], [321, 330, "USERNAME"], [311, 318, "TIME"], [287, 297, "USERNAME"]],
- "mberttokens"：文本对应的多语言BERT Token拆分结果
示例内容：
["Sub", "##ject", ":", "Group", "Mess", "##aging", "for", "Ad", "##mission", "##s", "Process", "Good", "morning", ",", "everyone", ",", "I", "hope", "this", "message", "finds", "you", "well", ".", "As", "we", "continue", "our", "admission", "##s", "processes", ",", "I", "would", "like", "to", "update", "you", "on", "the", "latest", "developments", "and", "key", "information", ".", "Please", "find", "below", "the", "time", "##line", "for", "our", "upcoming", "meetings", ":", "-", "w", "##yn", "##q", "##vr", "##h", "##0", "##53", "-", "Meeting", "at", "10", ":", "20", "##am", "-", "luka", ".", "bu", "##rg", "-", "Meeting", "at", "21", "-", "q", "##ahi", "##l", ".", "wit", "##tau", "##er", "-", "Meeting", "at", "quarter", "past", "13", "-", "gh", "##ola", "##mh", "##osse", "##in", ".", "rus", "##ch", "##ke", "-", "Meeting", "at", "9", ":", "47", "PM", "-", "p", "##d", "##m", "##jr", "##sy", "##oz", "##14", "##60"]
- "mbert_bio_labels"：基于多语言BERT Token的机器学习BIO标注任务标签
示例内容：
["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "B-USERNAME", "I-USERNAME", "I-USERNAME", "O", "O", "O", "O", "O", "O", "O", "B-TIME", "I-TIME", "I-TIME", "O", "B-USERNAME", "I-USERNAME", "O", "O", "O", "B-TIME", "I-TIME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "I-USERNAME", "O", "O", "O", "O", "B-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "I-TIME", "O", "B-USERNAME", "I-USERNAME"],"
- "id"：数据条目的唯一标识，用于后续参考与反馈
示例内容："40767A"
- "language"：文本所属语言
示例内容："English"
- "set"：机器学习数据集的划分类型（训练/验证）
示例内容："train"

*注：为最大化与各类软件的兼容性，嵌套对象均以字符串形式存储。

# 六、关于我们

Ai4Privacy团队致力于打造人工智能领域的21世纪全球安全安全带，助力防范个人信息被纳入数据管道所带来的潜在风险。

订阅通讯与获取最新动态：[www.Ai4Privacy.com](www.Ai4Privacy.com)
- 招聘机器学习工程师、开发人员、Beta测试人员、人机协同标注验证人员（支持多语言）
- 支持与现有开源解决方案集成
- 在Discord上向我们提问：[https://discord.gg/kxSbJrUQZF](https://discord.gg/kxSbJrUQZF)

# 七、路线图与未来规划

- 实现碳中和
- 新增命名实体识别（Named Entity Recognition，简称NER）基准测试方法
- 优化多语言支持，尤其是本地化适配
- 持续扩充训练与测试数据集规模

# 八、已知问题

存在标注误差，主要集中在较为宽泛的信息类型，如国家、时间与头衔。例如，瑞士国名、Dr.头衔偶尔可能无法被正确标注。我们计划在后续更新中减少此类标注错误。

# 九、应用场景

**聊天机器人**：在聊天机器人系统中集成PII掩码模型，可自动脱敏姓名、地址、电话号码、电子邮箱等敏感信息，保障用户对话的隐私与安全。
**客户支持系统**：在通过工单或在线聊天与客户交互时，PII掩码可帮助保护敏感客户数据，使支持人员能够在不泄露个人信息的前提下处理咨询。
**电子邮件过滤**：电子邮件服务商可借助PII掩码模型，自动检测并脱敏来往邮件中的PII，降低敏感信息意外泄露的风险。
**数据匿名化**：处理包含PII的大型数据集（如医疗或金融记录）的机构，可在共享数据用于研究、分析或协作前，借助PII掩码模型实现数据匿名化。
**社交媒体平台**：在社交媒体平台中集成PII掩码功能，可帮助用户保护个人信息免受未授权访问，打造更安全的线上环境。
**内容审核**：PII掩码可协助内容审核系统自动检测并模糊/脱敏用户生成内容中的敏感信息，避免个人细节被意外分享。
**在线表单**：通过在线表单（如注册表单或调查问卷）收集用户数据的Web应用，可使用PII掩码模型实时对收集到的信息进行匿名化或脱敏处理，提升隐私保护水平。
**协同文档编辑**：协同平台与文档编辑工具可在多用户编辑共享文档时，借助PII掩码模型自动脱敏敏感信息。
**研究与数据共享**：研究人员与机构在共享数据集用于协作、分析或发表时，可借助PII掩码模型保障数据的隐私性与机密性，降低数据泄露或身份盗用的风险。
**内容生成**：内容生成系统（如文章生成器或大语言模型）在创建示例文本时，可借助PII掩码自动脱敏或生成虚构PII，保护个人隐私。
(...and whatever else your creative mind can think of) → （以及其他所有你能想到的创意场景）

# 十、许可协议

鼓励学术使用，但需正确引用并遵守相关许可条款*。商业实体如需获取许可或额外数据访问权限，请联系[e-mail]。
*相关条款适用，完整细节请参阅[LICENSE.md](LICENSE.md)。

# 十一、支持与维护

AI4Privacy项目隶属于[Ai Suisse SA](https://www.aisuisse.com/)。

原始信息汇总

数据集概述

基本信息

• 数据集名称: Ai4Privacy PII 300k Dataset
• 数据集大小: 30.4m text tokens，约220,000+个示例，包含7.6m PII tokens
• 语言: 英语、法语、德语、意大利语、西班牙语、荷兰语
• 许可证: 其他，详细信息见license.md
• 源数据类型: 原始数据

数据集结构

• 数据文件配置:
  • 训练集: data/train/*.jsonl
  • 验证集: data/validation/*.jsonl

数据集内容

• 数据条目: 每个条目包含以下字段：
  • source_text: 包含PII的自然语言文本
  • target_text: 源文本的掩码版本
  • privacy_mask: PII的明确格式标签
  • span_labels: 文本中私人信息的精确映射范围
  • mberttokens: 文本的多语言BERT标记分解
  • mbert_bio_labels: 与BIO标记任务相关的标签
  • id: 条目的ID
  • language: 内容语言
  • set: 机器学习集类型

数据集用途

• 目的: 训练和评估模型以从文本中移除个人识别和敏感信息，特别适用于AI助手和LLMs。
• 应用场景:
  • 聊天机器人
  • 客户支持系统
  • 电子邮件过滤
  • 数据匿名化
  • 社交媒体平台
  • 内容审核
  • 在线表单
  • 协作文档编辑
  • 研究和数据共享
  • 内容生成

数据集特点

• 多语言支持: 支持6种语言，具有8个司法管辖区的强大本地化。
• 数据质量: 通过人类在环验证，具有高数据质量，QA结果透明。
• 数据生成: 使用专有算法生成合成数据，确保无隐私违规。
• 数据分割: 训练/验证分割比例为79% - 21%。

兼容的机器学习任务

• 任务类型:
  • 文本分类
  • 令牌分类
  • 表格问答
  • 问答
  • 零样本分类
  • 摘要
  • 特征提取
  • 文本生成
  • 文本到文本生成
  • 翻译
  • 填充掩码
  • 表格分类
  • 表格到文本
  • 文本检索
  • 其他

许可证和使用

• 学术使用: 鼓励学术使用，需提供适当引用并遵守类似许可条款。
• 商业使用: 商业实体应联系[e-mail]进行许可查询和额外数据访问。

支持和维护

• 项目关联: AI4Privacy项目与Ai Suisse SA关联。

搜集汇总

数据集介绍

构建方式

该数据集是通过合成数据生成的，使用了专有的算法来确保不会侵犯隐私。数据集中包含了约220,000个文本条目，共30.4m个文本token，其中有7.6m个PII token。数据集被分为两个子数据集：OpenPII-220k和FinPII-80k，其中FinPII包含了针对金融和保险的额外类别。数据集被分为训练集和验证集，比例为79%和21%。数据集还支持多种语言，包括英语、法语、德语、意大利语、荷兰语和西班牙语。

使用方法

使用该数据集的方法包括使用Python语言和HuggingFace的datasets库。用户可以通过pip安装datasets库，然后使用load_dataset函数来加载数据集。数据集中的每一行都是一个json对象，包含了自然语言文本和PII占位符。用户可以使用HuggingFace的transformers库中的模型来进行训练和评估。数据集还支持多种机器学习任务，包括token classification、text generation和translation等。

背景与挑战

背景概述

随着人工智能技术的发展，个人隐私保护已成为一个日益重要的议题。'ai4privacy/pii-masking-300k'数据集正是在此背景下创建的，旨在训练和评估模型以从文本中移除个人身份信息（PII）。该数据集由Ai4Privacy组织提供，创建于2023年，是目前世界上最大的开放PII数据集之一。它包含了220k个文本条目，涉及27种PII类别，涵盖了教育、健康和心理学等749个讨论主题。此外，FinPII子数据集还包含了针对保险和金融领域的约20种额外类别。该数据集支持6种语言，并在8个司法管辖区具有强大的本地化特性。数据集的创建时间、主要研究人员或机构、核心研究问题以及对相关领域的影响力，使其成为研究和开发隐私保护技术的宝贵资源。

当前挑战

尽管'ai4privacy/pii-masking-300k'数据集提供了丰富的数据资源，但仍然面临着一些挑战。首先，PII识别和掩码的准确性是一个关键问题，尤其是在处理非常粗略的信息类型，如国家、时间和标题时，可能会出现标签错误。其次，尽管数据集支持多语言，但本地化和语言之间的差异可能影响模型的泛化能力。此外，随着隐私保护法规的不断发展，如GDPR和AI法案，数据集的更新和维护也面临着挑战。最后，如何在保持模型性能的同时，确保数据集的隐私和安全，也是需要进一步研究和解决的问题。

常用场景

经典使用场景

该数据集被广泛应用于各类文本处理任务中，尤其适用于训练和评估模型以去除文本中的个人可识别和敏感信息。在人工智能助手和大型语言模型（LLMs）的背景下，该数据集对于保护用户隐私至关重要。

解决学术问题

该数据集解决了在文本中自动识别和遮蔽个人可识别信息（PII）的难题，这对于确保数据安全和用户隐私保护具有重要意义。它帮助研究人员和开发者构建更安全的AI系统，特别是在处理包含敏感信息的文本数据时。

实际应用

该数据集的实际应用场景包括但不限于聊天机器人、客户支持系统、电子邮件过滤、数据匿名化、社交媒体平台内容审核、在线表单处理、协作文档编辑以及研究数据共享等。

数据集最近研究

最新研究方向

在人工智能与隐私保护交叉领域，'ai4privacy/pii-masking-300k'数据集代表了隐私信息屏蔽技术的前沿。该数据集专注于从文本中移除个人可识别信息（PII），特别是在教育、健康和心理学等领域的讨论中。它涵盖了27种PII类别，针对749个讨论主题/用例，并包含约30.4m个文本标记和7.6m个PII标记，支持6种语言，具有高度本地化。数据集通过专有算法生成合成数据，确保无隐私侵犯，并通过人类在环验证，保证高质量。此外，数据集还分为OpenPII-220k和FinPII-80k两个子集，FinPII包括针对金融和保险的额外类别。该数据集支持多种机器学习任务，如标记分类、文本生成等，为隐私保护研究提供了宝贵的资源。