Abzalbek89/corpus_clean

---
language:
- kk
license: apache-2.0
task_categories:
- text-generation
- fill-mask
size_categories:
- 1M<n<10M
tags:
- kazakh
- corpus
- nlp
- language-modeling
- cleaned
dataset_info:
features:
- name: text
dtype: string
- name: source
dtype: string
splits:
- name: train
num_examples: 1502583
- name: validation
num_examples: 15177
---

# Kazakh Cleaned Corpus

Cleaned and deduplicated Kazakh-language text corpus built from multiple open sources. Designed for pretraining and fine-tuning Kazakh language models.

## Dataset Summary

| | Count |
|---|---|
| **Train** | 1,502,583 |
| **Validation** | 15,177 |
| **Total** | 1,517,760 |

The dataset was assembled from the [kz-transformers/multidomain-kazakh-dataset](https://huggingface.co/datasets/kz-transformers/multidomain-kazakh-dataset) collection and processed through a multi-stage cleaning pipeline.

## Sources

| Source | Raw texts | Clean texts | Description |
|---|---|---|---|
| `md_oscar` | 269,047 | 239,807 | OSCAR web crawl |
| `md_kazakhNews` | 2,107,986 | 129,349 | Kazakh news articles |
| `md_kazakhBooks` | 8,423 | 20,482* | Kazakh books |
| `md_leipzig` | 1,706,485 | 1,128,122 | Leipzig corpora collection |
| **Total** | **4,091,941** | **1,517,760** | |

\* Books are split into chunks of ≤50K characters, so the clean count exceeds raw count.

## Cleaning Pipeline

Each text passes through 9 sequential filters (ordered fast → slow):

1. **OSCAR dict fix** — unwraps texts stored as `{'text': '...'}` Python dict literals
2. **NFC normalization** — Unicode NFC, control character removal, whitespace normalization
3. **Minimum length** — ≥50 characters, ≥10 words
4. **Kazakh character check** — must contain at least one Kazakh-specific character (Ә, Ғ, Қ, Ң, Ө, Ұ, Ү, Һ, І, etc.)
5. **Script profile** — Cyrillic ≥60%, Latin ≤25%
6. **Junk filter** — URL density ≤5/1K chars, ≤5 HTML tags, special char ratio ≤40%, no boilerplate patterns
7. **Gzip repetition** — compression ratio ≥0.20 (filters repetitive/degenerate text)
8. **FastText LID** — language identification: `kk` confidence ≥0.50, gap to nearest rival ≥0.10
9. **Exact dedup** — MD5-based deduplication across all sources (1,033 duplicates removed)

### Rejection Statistics

| Reason | Count |
|---|---|
| `no_kaz_chars` | 1,888,456 |
| `too_few_words` | 323,914 |
| `too_short` | 255,993 |
| `lid_rejected` | 74,315 |
| `junk` | 45,297 |
| `script_profile` | 19,871 |
| `gzip_repetition` | 10,706 |
| `dedup` | 1,033 |

## Fields

- **`text`** (`string`) — cleaned text content
- **`source`** (`string`) — origin dataset identifier (e.g., `md_oscar`, `md_leipzig`)

## Usage

```python
from datasets import load_dataset

ds = load_dataset("Abzalbek89/corpus_clean")

# Access splits
train = ds["train"]
val = ds["validation"]

print(train[0]["text"][:200])
print(train[0]["source"])
```

## License

Apache 2.0

---
language:
- 哈萨克语（Kazakh）
license: apache-2.0
task_categories:
- 文本生成（text-generation）
- 掩码填充（fill-mask）
size_categories:
- 100万 < 样本数 < 1000万
tags:
- 哈萨克语（Kazakh）
- 语料库（corpus）
- 自然语言处理（NLP）
- 语言建模（language-modeling）
- 清洗后（cleaned）
dataset_info:
features:
- name: text
dtype: string
- name: source
dtype: string
splits:
- name: train
num_examples: 1502583
- name: validation
num_examples: 15177
---

# 清洗后哈萨克语语料库（Kazakh Cleaned Corpus）

本数据集为经清洗与去重的哈萨克语文本语料库，源自多个开源数据源，专为哈萨克语大语言模型（Large Language Model, LLM）的预训练与微调任务设计。

## 数据集概览

| 划分 | 样本数 |
|---|---|
| **训练集** | 1,502,583 |
| **验证集** | 15,177 |
| **总计** | 1,517,760 |

本数据集整合自[kz-transformers/multidomain-kazakh-dataset](https://huggingface.co/datasets/kz-transformers/multidomain-kazakh-dataset)数据集集合，并经过多阶段清洗流程处理。

## 数据源

| 数据源标识 | 原始文本数 | 清洗后文本数 | 说明 |
|---|---|---|---|
| `md_oscar` | 269,047 | 239,807 | OSCAR网页爬取数据 |
| `md_kazakhNews` | 2,107,986 | 129,349 | 哈萨克语新闻文章 |
| `md_kazakhBooks` | 8,423 | 20,482* | 哈萨克语书籍 |
| `md_leipzig` | 1,706,485 | 1,128,122 | 莱比锡语料库集合 |
| **总计** | **4,091,941** | **1,517,760** | |

* 注：书籍被切割为不超过50000字符的片段，因此清洗后文本数多于原始文本数。

## 清洗流程

每段文本将依次通过9级过滤（按处理速度从快到慢排序）：

1. **OSCAR格式修复**：解析以Python字典字面量`{'text': '...'}`格式存储的文本
2. **NFC归一化**：执行Unicode归一化格式C（NFC），移除控制字符并标准化空白字符
3. **最小长度限制**：文本长度≥50字符，且词数≥10
4. **哈萨克语字符校验**：必须包含至少一个哈萨克语专属字符（如Ә、Ғ、Қ、Ң、Ө、Ұ、Ү、Һ、І等）
5. **字符集占比校验**：西里尔字母占比≥60%，拉丁字母占比≤25%
6. **垃圾内容过滤**：URL密度≤5个/1000字符，HTML标签数≤5，特殊字符占比≤40%，且无模板化冗余内容
7. **Gzip重复度校验**：压缩比≥0.20（用于过滤重复或无效的退化文本）
8. **FastText语言识别**：语言识别结果为哈萨克语（`kk`）的置信度≥0.50，且与第二置信语言的得分差≥0.10
9. **精确去重**：基于MD5哈希的跨数据源去重（共移除1033条重复样本）

### 拒绝统计

| 拒绝原因 | 样本数 |
|---|---|
| 无哈萨克语专属字符 | 1,888,456 |
| 词数过少 | 323,914 |
| 长度过短 | 255,993 |
| 语言识别不通过 | 74,315 |
| 垃圾内容 | 45,297 |
| 字符集占比不达标 | 19,871 |
| 重复度过高 | 10,706 |
| 重复样本 | 1,033 |

## 数据字段

- **`text`**（字符串类型）：清洗后的文本内容
- **`source`**（字符串类型）：数据源标识（例如`md_oscar`、`md_leipzig`）

## 使用示例

python
from datasets import load_dataset

ds = load_dataset("Abzalbek89/corpus_clean")

# 加载数据集划分
train = ds["train"]
val = ds["validation"]

# 打印首条样本的前200个字符与数据源标识
print(train[0]["text"][:200])
print(train[0]["source"])

## 许可证

Apache 2.0