nileagi/swahili-language-exposure

--- license: mit language: - sw task_categories: - text-generation - question-answering - feature-extraction tags: - swahili - language-exposure - pretraining - low-resource-language - african-languages - conversational-text pretty_name: Swahili Language Exposure (NileAGI) size_categories: unknown configs: - config_name: default data_files: - split: train path: data/train-* - split: validation path: data/validation-* dataset_info: features: - name: text dtype: string splits: - name: train num_bytes: 3077404935 num_examples: 1589831 - name: validation num_bytes: 161180380 num_examples: 83676 download_size: 1936721770 dataset_size: 3238585315 --- # swahili-language-exposure ## Dataset Summary **swahili-language-exposure** is a large-scale Swahili (Kiswahili) corpus designed for **language exposure and continued pretraining** of language models. Unlike instruction-tuning datasets, this dataset focuses on exposing models to **natural Swahili usage** across conversations, explanations, narratives, technical discussions, and mixed-domain text. The goal is to improve **fluency, vocabulary coverage, syntax, and cultural grounding** in Swahili. This dataset is developed and maintained by **NileAGI**. --- ## Dataset Purpose This dataset is intended for: * Continued pretraining (DAPT / CPT) * Language exposure before instruction tuning * Improving Swahili fluency and coherence * Reducing English dominance in multilingual models * Low-resource language research It is **not optimized for instruction-following by default**. --- ## Dataset Format The dataset is provided in **JSON Lines (`.jsonl`)** format. Each line contains a **single Swahili text sample**, without enforced instruction–response structure: ```json { "text": "Maudhui ya asili kwa Kiswahili yaliyoandikwa au kuzungumzwa katika mazingira halisi." } ```` ### Example ```json {"text":"Nilikuwa najifunza kuhusu mitandao ya neva na jinsi inavyotumika katika utambuzi wa picha, lakini changamoto kubwa ilikuwa kupata data iliyosawazika."} ``` --- ## Supported Tasks * Language modeling * Text generation * Conversational fluency improvement * Vocabulary and grammar learning * Domain adaptation for Swahili * Foundation training for downstream fine-tuning --- ## Language * **Swahili (Kiswahili)** — primary language (`sw`) * Natural code-switching with English technical terms may appear * Informal and semi-formal registers are both present --- ## Data Sources The dataset was curated from: * Educational explanations and mentoring sessions * Informal dialogues and narrative text * Mixed-domain Swahili content reflecting real usage All data has been anonymized and cleaned. --- ## Data Preprocessing The following preprocessing steps were applied: * Filtering of extremely short or low-quality text * Normalization of whitespace and encoding * Preservation of natural language flow and code-switching No artificial instruction templates were added. --- ## Intended Use ### Primary Use * Continued pretraining (CPT/DAPT) for Swahili * Language exposure before instruction tuning * Improving multilingual model balance ### Secondary Use * Linguistic analysis * Swahili NLP benchmarking * Data augmentation for low-resource research --- ## Out-of-Scope Uses This dataset is **not intended** for: * Direct instruction fine-tuning (see companion instruct dataset) * Preference learning or RLHF * Surveillance, profiling, or harmful content generation --- ## Biases and Limitations * Informal conversational Swahili is more common than formal prose * Technical domains may be overrepresented * Regional phrasing reflects contributor backgrounds Users should consider complementary corpora for broader coverage. --- ## Ethical Considerations * Personally identifiable information has been removed or masked * No sensitive personal attributes are intentionally included * Released strictly for responsible AI research --- ## License This dataset is released under the **MIT License**. --- ## Citation If you use this dataset, please cite: ```bibtex @dataset{swahili_language_exposure, title = {swahili-language-exposure: A Swahili Language Exposure Dataset}, author = {NileAGI}, year = {2026}, publisher = {Hugging Face} } ```许可证：MIT许可证 语言： - sw 任务类别： - 文本生成 - 问答 - 特征提取 标签： - 斯瓦西里语 - 语言接触预训练 - 低资源语言 - 非洲语言 - 会话文本 美观名称：斯瓦西里语语言接触数据集（NileAGI） 规模类别：未知 配置： - 配置名称：default 数据文件： - 拆分集：train 路径：data/train-* - 拆分集：validation 路径：data/validation-* 数据集信息： 特征： - 名称：text 数据类型：字符串 拆分集： - 名称：train 字节数：3077404935 样本数：1589831 - 名称：validation 字节数：161180380 样本数：83676 下载大小：1936721770 数据集总大小：3238585315 # 斯瓦西里语语言接触数据集（swahili-language-exposure） ## 数据集摘要 **swahili-language-exposure** 是一个大规模斯瓦西里语（Kiswahili）语料库，专为语言模型的**语言接触与持续预训练**设计。与指令微调数据集不同，本数据集专注于让模型接触**覆盖会话、解释、叙事、技术讨论及混合领域文本的自然斯瓦西里语使用场景**，旨在提升模型在斯瓦西里语上的**流畅度、词汇覆盖范围、句法能力与文化适配性**。本数据集由**NileAGI**开发并维护。 ## 数据集用途 本数据集适用于以下场景： * 持续预训练（领域自适应预训练DAPT / 持续预训练CPT） * 指令微调前的语言接触训练 * 提升斯瓦西里语流畅度与连贯性 * 降低多语言模型中的英语主导性 * 低资源语言研究 默认情况下，本数据集未针对指令跟随任务优化。 ## 数据集格式 本数据集以**JSON Lines（.jsonl）**格式提供。每行包含**单条斯瓦西里语文本样本**，无强制的指令-回复结构： json { "text": "Maudhui ya asili kwa Kiswahili yaliyoandikwa au kuzungumzwa katika mazingira halisi." } ### 示例 json {"text":"Nilikuwa najifunza kuhusu mitandao ya neva na jinsi inavyotumika katika utambuzi wa picha, lakini changamoto kubwa ilikuwa kupata data iliyosawazika."} ## 支持任务 * 语言建模 * 文本生成 * 会话流畅度提升 * 词汇与语法学习 * 斯瓦西里语领域自适应 * 下游微调的基础预训练 ## 语言 * **斯瓦西里语（Kiswahili）** — 核心语言（代码标识：`sw`） * 可能出现自然混入英语技术术语的语码转换现象 * 同时包含非正式与半正式语体 ## 数据来源 本数据集整理自以下内容： * 教育性解释与辅导会话 * 非正式对话与叙事文本 * 反映真实使用场景的混合领域斯瓦西里语内容 所有数据均已完成匿名化与清洗处理。 ## 数据预处理 已应用以下预处理步骤： * 过滤极短文本与低质量内容 * 标准化空白符与编码格式 * 保留自然语言流与语码转换现象 未添加人工指令模板。 ## 预期用途 ### 主要用途 * 斯瓦西里语持续预训练（CPT/DAPT） * 指令微调前的语言接触训练 * 优化多语言模型的语言平衡性 ### 次要用途 * 语言学分析 * 斯瓦西里语自然语言处理基准测试 * 低资源语言研究的数据增强 ## 不适用场景 本数据集**不适用**于以下场景： * 直接指令微调（请参阅配套的指令数据集） * 偏好学习或基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF） * 监控、用户画像或有害内容生成 ## 偏差与局限性 * 非正式会话斯瓦西里语相较于正式散文占比更高 * 技术领域内容可能占比过高 * 区域用语风格反映了贡献者的背景信息 使用者应考虑补充其他语料库以获得更广泛的覆盖范围。 ## 伦理考量 * 已移除或掩码处理所有个人可识别信息 * 未故意包含任何敏感个人属性 * 仅用于负责任的人工智能研究 ## 许可证 本数据集采用**MIT许可证**发布。 ## 引用方式 如果使用本数据集，请引用以下文献： bibtex @dataset{swahili_language_exposure, title = {swahili-language-exposure: A Swahili Language Exposure Dataset}, author = {NileAGI}, year = {2026}, publisher = {Hugging Face} }