Zarinaaa/commoncrawl_dataset

--- license: cc0-1.0 tags: - kyrgyz - common-crawl - web-scraping - text-corpus - low-resource-languages - nlp - turkic-languages language: - ky task_categories: - text-generation - fill-mask - text-classification size_categories: - 100M<n<1B pretty_name: Kyrgyz CommonCrawl Text Corpus --- # Kyrgyz CommonCrawl Dataset A **271 MB** text corpus of Kyrgyz language data extracted from [CommonCrawl](https://commoncrawl.org/) — one of the largest openly available Kyrgyz text collections for NLP research. --- ## Dataset Description This dataset contains Kyrgyz-language web text scraped from CommonCrawl archives, filtered by the Kyrgyz language tag (`ky`). The data covers a wide range of domains including news, blogs, government sites, educational content, and general web pages. **Why this matters:** Kyrgyz is a low-resource Turkic language spoken by ~7 million people. High-quality text corpora are essential for training language models, yet very few large-scale Kyrgyz datasets exist publicly. --- ## Dataset Summary | Property | Value | |----------|-------| | **Total size** | 271 MB | | **Language** | Kyrgyz (ky) | | **Format** | CSV | | **Source** | CommonCrawl (filtered by `ky` language tag) | | **Files** | 31 CSV files | | **License** | CC0 (public domain) | --- ## File Structure | File | Size | Description | |------|------|-------------| | `data_MN.csv` | 29.6 MB | Large text segment | | `data.csv` | 9.98 MB | General web text | | `data_bilesinbi.csv` | 7.11 MB | Domain-specific data | | `Merged file2.csv` | 1.66 MB | Merged text segments | | `data_f.csv` | 703 kB | Filtered subset | | `8april_final.csv` | 634 kB | Cleaned snapshot | | `data.numbers` | 581 kB | Statistics/metadata | | `bia.csv` | 37.6 kB | Small subset | | `data_ecoproduct.csv` | 19.9 kB | Eco/product domain | | ... | ... | Additional CSV files | --- ## Use Cases - **Language model pretraining** — Training or fine-tuning LLMs for Kyrgyz (e.g., GPT, BERT, LLaMA) - **Text classification** — Building Kyrgyz text classifiers - **Machine translation** — Source data for Kyrgyz ↔ other language pairs - **Linguistic research** — Studying modern Kyrgyz web language usage - **Punctuation / grammar models** — Training data for text normalization tools - **NER & information extraction** — Building Kyrgyz entity recognizers --- ## Data Collection The data was collected by: 1. Querying CommonCrawl archives for pages tagged with the Kyrgyz language identifier (`ky`) 2. Extracting text content from the matched web pages 3. Cleaning and organizing into CSV format 4. Deduplication and quality filtering --- ## Preprocessing Recommendations Before using this dataset, consider: - **Deduplication** — Web-crawled data often contains duplicate paragraphs across pages - **Language verification** — Some pages may contain mixed-language content (Kyrgyz + Russian is common) - **Quality filtering** — Remove boilerplate (navigation menus, footers, cookie notices) - **Encoding normalization** — Ensure consistent Cyrillic encoding (UTF-8) --- ## Limitations - **Web-crawled data** may contain noise, boilerplate HTML artifacts, and mixed-language content - **No manual curation** — quality varies across files - **Potential duplicates** across different CSV files - **Bias toward web-present content** — overrepresentation of news and government text, underrepresentation of informal speech --- ## Related Resources - 🤗 [Kyrgyz Punctuation Model](https://huggingface.co/Zarinaaa/punctuator_model) — Trained using data from this corpus - 🤗 [Kyrgyz Morphological Analysis](https://huggingface.co/Zarinaaa/morphological_analysis) — BERT-based morphological tagger --- ## Citation ```bibtex @dataset{uvalieva2024kyrgyz_commoncrawl, author = {Uvalieva, Zarina}, title = {Kyrgyz CommonCrawl Text Corpus}, year = {2024}, url = {https://huggingface.co/datasets/Zarinaaa/commoncrawl_dataset} } ``` --- ## Author **Zarina Uvalieva** — ML Engineer specializing in NLP for low-resource languages. - 🤗 [HuggingFace](https://huggingface.co/Zarinaaa)--- 许可证：CC0-1.0 标签： - 吉尔吉斯语 - CommonCrawl - 网页抓取 - 文本语料库 - 低资源语言（low-resource languages） - 自然语言处理（Natural Language Processing，简称NLP） - 突厥语族语言 语言： - ky 任务类别： - 文本生成 - 掩码填充 - 文本分类 规模类别： - 100M<n<1B 展示名称：吉尔吉斯语CommonCrawl文本语料库 --- # 吉尔吉斯语CommonCrawl数据集 **271 MB**的吉尔吉斯语文本语料库，数据源自CommonCrawl——目前公开可用的规模最大的吉尔吉斯语文本语料库之一，适用于自然语言处理研究。 --- ## 数据集描述 本数据集包含从CommonCrawl档案中抓取的吉尔吉斯语文本，通过吉尔吉斯语语言标签（`ky`）进行过滤。数据涵盖新闻、博客、政府网站、教育内容及通用网页等多个领域。 **重要意义**：吉尔吉斯语是约700万使用者的低资源突厥语族语言。高质量文本语料库是训练语言模型的核心基础，但目前公开可用的大规模吉尔吉斯语数据集极为稀缺。 --- ## 数据集概览 | 属性 | 取值 | |----------|-------| | **总大小** | 271 MB | | **语言** | 吉尔吉斯语（ky） | | **格式** | CSV | | **来源** | CommonCrawl（通过`ky`语言标签过滤） | | **文件数量** | 31个CSV文件 | | **许可证** | CC0（公共领域） | --- ## 文件结构 | 文件名称 | 大小 | 描述 | |------|------|-------------| | `data_MN.csv` | 29.6 MB | 大型文本片段 | | `data.csv` | 9.98 MB | 通用网页文本 | | `data_bilesinbi.csv` | 7.11 MB | 特定领域数据 | | `Merged file2.csv` | 1.66 MB | 合并后的文本片段 | | `data_f.csv` | 703 kB | 过滤后的子集 | | `8april_final.csv` | 634 kB | 清理后的快照 | | `data.numbers` | 581 kB | 统计数据/元数据 | | `bia.csv` | 37.6 kB | 小型子集 | | `data_ecoproduct.csv` | 19.9 kB | 生态/产品领域数据 | | `...` | `...` | 其他CSV文件 | --- ## 应用场景 - **语言模型预训练**：用于吉尔吉斯语大语言模型（LLM）的训练或微调（如GPT、BERT、LLaMA） - **文本分类**：构建吉尔吉斯语文本分类器 - **机器翻译**：作为吉尔吉斯语与其他语言互译的源数据 - **语言学研究**：研究现代吉尔吉斯语网页语言使用情况 - **标点与语法模型**：作为文本归一化工具的训练数据 - **命名实体识别与信息抽取**：构建吉尔吉斯语实体识别系统 --- ## 数据收集 本数据集通过以下步骤收集： 1. 在CommonCrawl档案中检索带有吉尔吉斯语语言标识符`ky`的网页 2. 从匹配到的网页中提取文本内容 3. 清理并整理为CSV格式 4. 执行去重与质量过滤 --- ## 预处理建议 使用本数据集前，建议进行以下处理： - **去重**：网页抓取数据通常存在跨页面的重复段落 - **语言验证**：部分页面可能包含混合语言内容（吉尔吉斯语+俄语较为常见） - **质量过滤**：移除模板内容（导航菜单、页脚、Cookie提示等） - **编码归一化**：确保使用统一的西里尔字母UTF-8编码 --- ## 局限性 - **网页抓取数据**可能包含噪声、HTML模板残留及混合语言内容 - **无人工标注环节**：不同文件的质量存在差异 - **不同CSV文件间可能存在重复数据** - **内容偏向公开网页内容**：新闻与政府文本占比偏高，非正式口语内容占比偏低 --- ## 相关资源 - 🤗 [吉尔吉斯语标点模型](https://huggingface.co/Zarinaaa/punctuator_model)——基于本语料库训练得到 - 🤗 [吉尔吉斯语形态分析工具](https://huggingface.co/Zarinaaa/morphological_analysis)——基于BERT的形态标注器 --- ## 引用 bibtex @dataset{uvalieva2024kyrgyz_commoncrawl, author = {Uvalieva, Zarina}, title = {Kyrgyz CommonCrawl Text Corpus}, year = {2024}, url = {https://huggingface.co/datasets/Zarinaaa/commoncrawl_dataset} } --- ## 作者 **扎丽娜·乌瓦列耶娃（Zarina Uvalieva）**——专注于低资源语言自然语言处理的机器学习工程师。 - 🤗 [HuggingFace主页](https://huggingface.co/Zarinaaa)