inmoonlight/kor_hate

---
annotations_creators:
- crowdsourced
- expert-generated
language_creators:
- found
language:
- ko
license:
- cc-by-sa-4.0
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- multi-label-classification
paperswithcode_id: korean-hatespeech-dataset
pretty_name: Korean HateSpeech Dataset
dataset_info:
features:
- name: comments
dtype: string
- name: contain_gender_bias
dtype:
class_label:
names:
'0': 'False'
'1': 'True'
- name: bias
dtype:
class_label:
names:
'0': none
'1': gender
'2': others
- name: hate
dtype:
class_label:
names:
'0': hate
'1': offensive
'2': none
splits:
- name: train
num_bytes: 983608
num_examples: 7896
- name: test
num_bytes: 58913
num_examples: 471
download_size: 968449
dataset_size: 1042521
---

# Dataset Card for [Dataset Name]

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [Korean HateSpeech Dataset](https://github.com/kocohub/korean-hate-speech)
- **Repository:** [Korean HateSpeech Dataset](https://github.com/kocohub/korean-hate-speech)
- **Paper:** [BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection](https://arxiv.org/abs/2005.12503)
- **Point of Contact:** [Steven Liu]([e-mail])

### Dataset Summary

The Korean HateSpeech Dataset is a dataset of 8367 human-labeled entertainment news comments from a popular Korean news aggregation platform. Each comment was evaluated for either social bias (labels: `gender`, `others` `none`), hate speech (labels: `hate`, `offensive`, `none`) or gender bias (labels: `True`, `False`). The dataset was created to support the identification of toxic comments on online platforms where users can remain anonymous.

### Supported Tasks and Leaderboards

* `multi-label classification`: The dataset can be used to train a model for hate speech detection. A BERT model can be presented with a Korean entertainment news comment and be asked to label whether it contains social bias, gender bias and hate speech. Users can participate in a Kaggle leaderboard [here](https://www.kaggle.com/c/korean-hate-speech-detection/overview).

### Languages

The text in the dataset is in Korean and the associated is BCP-47 code is `ko-KR`.

## Dataset Structure

### Data Instances

An example data instance contains a `comments` containing the text of the news comment and then labels for each of the following fields: `contain_gender_bias`, `bias` and `hate`.

```python
{'comments':'설마 ㅈ 현정 작가 아니지??'
'contain_gender_bias': 'True',
'bias': 'gender',
'hate': 'hate'
}
```

### Data Fields

* `comments`: text from the Korean news comment
* `contain_gender_bias`: a binary `True`/`False` label for the presence of gender bias
* `bias`: determines the type of social bias, which can be:
* `gender`: if the text includes bias for gender role, sexual orientation, sexual identity, and any thoughts on gender-related acts
* `others`: other kinds of factors that are considered not gender-related but social bias, including race, background, nationality, ethnic group, political stance, skin color, religion, handicaps, age, appearance, richness, occupations, the absence of military service experience
* `none`: a comment that does not incorporate the bias
* `hate`: determines how aggressive the comment is, which can be:
* `hate`: if the text is defined as an expression that display aggressive stances towards individuals/groups with certain characteristics (gender role, sexual orientation, sexual identity, any thoughts on gender-related acts, race, background, nationality, ethnic group, political stance, skin color, religion, handicaps, age, appearance, richness, occupations, the absence of military service experience, etc.)
* `offensive`: if the text contains rude or aggressive contents, can emit sarcasm through rhetorical question or irony, encompass an unethical expression or conveys unidentified rumors
* `none`: a comment that does not incorporate hate

### Data Splits

The data is split into a training and development (test) set. It contains 8371 annotated comments that are split into 7896 comments in the training set and 471 comments in the test set.

## Dataset Creation

### Curation Rationale

The dataset was created to provide the first human-labeled Korean corpus for toxic speech detection from a Korean online entertainment news aggregator. Recently, two young Korean celebrities suffered from a series of tragic incidents that led to two major Korean web portals to close the comments section on their platform. However, this only serves as a temporary solution, and the fundamental issue has not been solved yet. This dataset hopes to improve Korean hate speech detection.

### Source Data

#### Initial Data Collection and Normalization

A total of 10.4 million comments were collected from an online Korean entertainment news aggregator between Jan. 1, 2018 and Feb. 29, 2020. 1,580 articles were drawn using stratified sampling and the top 20 comments were extracted ranked in order of their Wilson score on the downvote for each article. Duplicate comments, single token comments and comments with more than 100 characters were removed (because they could convey various opinions). From here, 10K comments were randomly chosen for annotation.

#### Who are the source language producers?

The language producers are users of the Korean online news platform between 2018 and 2020.

### Annotations

#### Annotation process

Each comment was assigned to three random annotators to assign a majority decision. For more ambiguous comments, annotators were allowed to skip the comment. See Appendix A in the [paper](https://arxiv.org/pdf/2005.12503.pdf) for more detailed guidelines.

#### Who are the annotators?

Annotation was performed by 32 annotators, consisting of 29 annotators from the crowdsourcing platform DeepNatural AI and three NLP researchers.

### Personal and Sensitive Information

[N/A]

## Considerations for Using the Data

### Social Impact of Dataset

The purpose of this dataset is to tackle the social issue of users creating toxic comments on online platforms. This dataset aims to improve detection of toxic comments online.

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

This dataset is curated by Jihyung Moon, Won Ik Cho and Junbum Lee.

### Licensing Information

[N/A]

### Citation Information

```
@inproceedings
{moon-et-al-2020-beep
title = "{BEEP}! {K}orean Corpus of Online News Comments for Toxic Speech Detection",
author = "Moon, Jihyung and
Cho, Won Ik and
Lee, Junbum",
booktitle = "Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.socialnlp-1.4",
pages = "25--31",
abstract = "Toxic comments in online platforms are an unavoidable social issue under the cloak of anonymity. Hate speech detection has been actively done for languages such as English, German, or Italian, where manually labeled corpus has been released. In this work, we first present 9.4K manually labeled entertainment news comments for identifying Korean toxic speech, collected from a widely used online news platform in Korea. The comments are annotated regarding social bias and hate speech since both aspects are correlated. The inter-annotator agreement Krippendorff{'}s alpha score is 0.492 and 0.496, respectively. We provide benchmarks using CharCNN, BiLSTM, and BERT, where BERT achieves the highest score on all tasks. The models generally display better performance on bias identification, since the hate speech detection is a more subjective issue. Additionally, when BERT is trained with bias label for hate speech detection, the prediction score increases, implying that bias and hate are intertwined. We make our dataset publicly available and open competitions with the corpus and benchmarks.",
}
```

### Contributions

Thanks to [@stevhliu](https://github.com/stevhliu) for adding this dataset.

原始信息汇总

数据集概述

基本信息

• 名称: Korean HateSpeech Dataset
• 语言: 韩语 (ko)
• 许可证: CC-BY-SA-4.0
• 多语言性: 单语种
• 大小: 1K<n<10K
• 源数据: 原始数据
• 任务类别: 文本分类
• 任务ID: 多标签分类

数据集内容

• 特征:
  • comments: 新闻评论文本，数据类型为字符串。
  • contain_gender_bias: 是否包含性别偏见，数据类型为类别标签，选项为False和True。
  • bias: 社会偏见类型，数据类型为类别标签，选项为none, gender, others。
  • hate: 评论的攻击性，数据类型为类别标签，选项为hate, offensive, none。
• 数据分割:
  • train: 训练集，包含7896个样本，总字节数为983608。
  • test: 测试集，包含471个样本，总字节数为58913。

数据集创建

• 注释创建者: 众包和专家生成
• 语言创建者: 发现
• 采集理由: 为了提供首个用于检测韩国在线娱乐新闻中毒素言论的人工标注语料库。
• 源数据收集: 从2018年至2020年，从韩国在线娱乐新闻聚合器收集了1040万条评论，通过分层抽样选出1580篇文章，并从中提取前20条评论进行标注。
• 注释过程: 每个评论由三名随机注释者进行标注，以达成多数决策。

使用考虑

• 社会影响: 旨在解决在线平台用户发布毒素评论的社会问题，提高在线毒素评论的检测能力。

附加信息

• 数据集维护者: Jihyung Moon, Won Ik Cho, Junbum Lee
• 贡献者: @stevhliu

搜集汇总

数据集介绍

构建方式

在互联网匿名评论生态中，有害言论的自动检测成为自然语言处理领域的重要课题。该数据集源自韩国主流新闻聚合平台2018年至2020年间约1040万条娱乐新闻评论，通过分层抽样选取1580篇文章，依据威尔逊评分提取每篇文章获赞与踩数最高的前20条评论，并剔除重复、单字符及超长文本。最终从筛选结果中随机抽取1万条评论，由32名标注者（含29名众包平台标注员与3名NLP研究员）进行三人交叉标注，以多数决原则确定标签，针对模糊样本允许跳过，构建了包含8367条标注样本的韩语有害言论语料库。

特点

该数据集以多标签分类为设计核心，每条评论同时标注三类属性：社会偏见（含性别偏见、其他偏见、无偏见）、仇恨言论（含仇恨、攻击性、无仇恨）及性别偏见存在性（是/否）。标签体系兼顾社会偏见与攻击性表达的内在关联，其中偏见类别覆盖性别角色、性取向、种族、政治立场、外貌等多元维度，仇恨等级则区分直接攻击与讽刺性冒犯。数据集划分为7896条训练样本与471条测试样本，支持基于BERT等预训练模型的细粒度毒性检测研究。

使用方法

研究者可通过HuggingFace数据集加载工具直接调用该资源，使用`load_dataset("inmoonlight/kor_hate")`获取训练集与测试集。每条数据以字典形式提供`comments`字段（韩语评论原文）及三个分类标签：`contain_gender_bias`（布尔值）、`bias`（三分类）、`hate`（三分类）。适用于多标签文本分类任务，可基于Kaggle竞赛平台（korean-hate-speech-detection）进行模型性能基准测试，推荐采用预训练语言模型（如BERT）联合学习偏见与仇恨标签以提升检测效果。

背景与挑战

背景概述

该数据集由Jihyung Moon、Won Ik Cho和Junbum Lee于2020年创建，源自韩国主流新闻聚合平台上的娱乐新闻评论，旨在应对匿名网络环境中日益严重的恶意言论问题。研究团队从2018年至2020年间收集的1040万条评论中，通过分层抽样和威尔逊得分排序，精选出8367条人工标注的评论，构建了首个韩语恶意言论检测语料库。该数据集聚焦于社会偏见（如性别、种族等）与仇恨言论的多标签分类，其发布直接回应了韩国网络平台因名人悲剧事件关闭评论区所暴露的深层社会矛盾，为自然语言处理领域提供了针对韩语语境下毒性言论检测的关键基准资源，推动了相关模型（如BERT）在跨语言恶意内容识别中的性能提升。

当前挑战

该数据集面临的核心挑战在于韩语在线评论中仇恨言论与偏见的复杂交织性。领域层面，恶意言论检测本身具有主观性，标注者间一致性（Krippendorff's alpha仅0.492-0.496）反映了定义模糊的困境，例如讽刺、反语等隐晦表达难以被准确归类。构建过程中，从1040万条原始评论筛选至1万条候选样本时，需剔除重复、单标记及超长评论（>100字符），导致数据规模受限（仅8367条标注样本），可能降低模型泛化能力。此外，32名标注员（含29名众包人员）对性别偏见、社会偏见等标签的判别标准差异，以及娱乐新闻领域特有的语境依赖性，进一步加剧了标注噪声，使得模型在区分“冒犯性”与“仇恨”类别时面临精度瓶颈。

常用场景

经典使用场景

在自然语言处理领域，该数据集最经典的使用场景是作为多标签分类任务的基准，用于训练模型同时检测在线评论中是否包含社会偏见、性别偏见以及仇恨言论。研究者通过将韩语娱乐新闻评论输入BERT等预训练语言模型，输出三个维度的标签，从而实现对有毒言论的细粒度识别。该场景尤其适用于匿名网络环境下的言论审核，为构建更安全的线上社区提供了数据支撑。

解决学术问题

该数据集解决了学术界在韩语仇恨言论检测领域缺乏高质量人工标注语料的关键问题。此前，相关研究多集中于英语、德语等语言，韩语资源极为匮乏。通过提供8367条经过多人一致性验证的评论，研究者得以系统探究社会偏见与仇恨言论之间的内在关联，实验表明联合学习偏见标签可提升仇恨言论检测性能，从而揭示了两种现象相互缠绕的深层机制，为跨语言有毒言论检测理论提供了重要实证。

衍生相关工作

该数据集衍生了一系列经典工作，包括KcELECTRA等韩语专用预训练模型在仇恨言论任务上的微调与评测，以及Kaggle平台发起的韩语仇恨言论检测竞赛，吸引了全球团队探索数据增强、对抗训练等优化策略。此外，研究者基于该语料提出了跨任务迁移学习框架，验证了偏见标签作为辅助任务对提升仇恨分类鲁棒性的有效性，推动了多任务学习在有害内容识别领域的理论发展。