argoveziriii/farsi-asr-unified-cleaned

Farsi ASR统一数据集（Parquet分片版）是一个大规模、高质量、完全标准化的波斯语（Farsi）语音到文本数据集合，专为现代机器学习和自动语音识别（ASR）工作流设计。该数据集整合了来自多个开放源的音频-文本对，并应用了严格的清理和标准化流程，所有数据以Parquet分片格式高效存储，内嵌音频。数据集总时长约1,392.8小时，总样本数为1,278,937个，音频格式为16 kHz。亮点包括高性能（Parquet分片约500 MB，支持快速读取）、自包含（每个Parquet文件包含文本和音频字节，无需单独管理音频文件夹）、流式友好（优化用于Hugging Face datasets库，可直接加载或流式传输）以及干净简洁（仅包含数据分片和README）。局限性包括：由于聚合多个大型网络抓取数据集，极少数样本可能存在语言不匹配（如阿拉伯语音频与波斯语转录），错误率估计低于0.01%；数字表达未完全标准化（可能使用波斯数字或拼写单词）。数据集结构包含path（原始音频文件路径）、text（清理和标准化后的转录文本）、audio（音频原始字节，16kHz、单声道、PCM WAV格式）和sampling_rate（采样率，固定为16000）等字段。处理流程包括音频标准化、初始去重、高级文本规范化（如标点标准化、字符白名单过滤、空白规范化）以及过滤（如移除空转录、短于1秒的音频、超过30秒的音频、字符率异常样本）。数据源包括CommonVoice_22_fa、srezas/farsi_voice_dataset等多个公开数据集。该数据集为训练和评估波斯语ASR模型提供了干净、统一、高性能的基础。

The Farsi ASR Unified Dataset (Parquet Sharded Edition) is a large-scale, high-quality, and fully standardized collection of Persian (Farsi) speech-to-text data, designed specifically for modern machine learning and ASR (Automatic Speech Recognition) workflows. It consolidates audio-text pairs from multiple open sources, applies a rigorous cleaning and normalization pipeline, and stores everything efficiently in Parquet shards with embedded audio. The total duration is approximately 1,392.8 hours, with 1,278,937 total samples, in Parquet format (sharded, with embedded 16 kHz audio). Highlights include high performance (Parquet shards of ~500 MB enable fast reads), self-contained design (each Parquet file includes both text and audio bytes, eliminating the need for separate audio folders), streaming-friendly optimization (for the Hugging Face datasets library, allowing direct loading or streaming without full download), and clean minimalism (only data shards and README). Limitations include: due to aggregating multiple large, web-scraped datasets, a very small fraction of samples may contain language mismatches (e.g., Arabic audio with Farsi transcript), with an estimated error rate below 0.01%; numeric expressions are not fully standardized (may appear as Persian digits or spelled-out words). The dataset structure includes columns: path (original relative audio file path), text (cleaned and normalized transcript), audio (raw bytes of audio in 16kHz, mono, PCM WAV format), and sampling_rate (consistently 16000). The processing pipeline involves audio standardization, initial deduplication, advanced text normalization (e.g., punctuation standardization, character whitelist filtering, whitespace normalization), and filtering (e.g., removal of empty transcripts, audio clips <1 second, duration >30 seconds, abnormal character rates). Source datasets include CommonVoice_22_fa, srezas/farsi_voice_dataset, and others. This dataset provides a clean, unified, and high-performance foundation for training and evaluating Farsi ASR models.