juanquivilla/sotto-transcript-cleanup

---
license: mit
task_categories:
- text-generation
language:
- en
size_categories:
- 100K<n<1M
tags:
- speech-to-text
- transcript-cleanup
- disfluency-correction
- synthetic-data
- sotto-asr
pretty_name: SottoASR Transcript Cleanup Dataset
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
dataset_info:
features:
- name: input
dtype: string
- name: output
dtype: string
splits:
- name: train
num_bytes: 33192538
num_examples: 135503
- name: validation
num_bytes: 1296731
num_examples: 6921
download_size: 18979669
dataset_size: 34489269
---

# SottoASR Transcript Cleanup Dataset

<p align="center">
<a href="https://sotto.app">sotto.app</a> ·
<a href="https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m">Trained Model (bf16)</a> ·
<a href="https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit">MLX 5-bit Model</a>
</p>

## Overview

124K+ synthetic training pairs for fine-tuning small language models on speech-to-text transcript cleanup. This dataset was used to train the [SottoASR transcript cleanup model](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m) — a 350M parameter model that **exceeds a prompted 2B model** on this task while being 8x faster.

Part of [**SottoASR**](https://sotto.app) — a local, privacy-first speech-to-text application for macOS.

## Task

**Input:** Raw, lowercase, unpunctuated ASR transcript with speech disfluencies
**Output:** Clean, properly formatted text with disfluencies removed

```jsonl
{"input": "uh the server is uh running low on memory", "output": "The server is running low on memory."}
{"input": "use redis wait no memcached is better", "output": "Use Memcached."}
{"input": "ship it", "output": "Ship it."}
{"input": "send the email to john period", "output": "Send the email to John."}
```

## Categories

| Category | % | Description |
|----------|---|-------------|
| self_correction | 14% | Speaker corrects themselves mid-sentence |
| preserve_wording | 13% | Clean input — model must NOT over-edit |
| filler_removal | 11% | Remove uh, um, uhm, er, ah |
| mixed | 10% | Multiple disfluency types combined |
| crutch_words | 8% | Remove basically, you know, I mean, etc. |
| false_start | 8% | Remove abandoned sentence beginnings |
| dictation_commands | 8% | Convert "period" → ".", "comma" → "," |
| misheard_words | 7% | Fix ASR errors (post gress → Postgres) |
| grammar | 7% | Fix spoken grammar (gonna → going to) |
| list_formatting | 6% | Convert spoken lists to numbered format |
| adversarial | 5% | Words that look like fillers but are meaningful |

## Domains

Software engineering (24%), general business (19%), casual conversation (15%), medical (10%), legal (8%), finance (7%), technical (5%), creative (5%), academic (5%)

## Generation Method

Three-layer approach:
1. **Programmatic corruption** (Layer 1) — deterministic disfluency injection into clean public text
2. **LLM-generated** (Layer 2) — context-dependent patterns via Qwen3.5-35B and Grok 4.20
3. **Hand-crafted** (Layer 3) — expert-written samples for edge cases

94.6% validation pass rate. Details in the [training research document](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m).

## Splits

| Split | Samples |
|-------|---------|
| train | 118,069 |
| val | 6,215 |

## License

MIT

license: MIT
task_categories:
- 文本生成
language:
- 英语
size_categories:
- 100K < n < 1M
tags:
- 语音转文字（speech-to-text）
- 转录清理（transcript-cleanup）
- 语病修正（disfluency-correction）
- 合成数据（synthetic-data）
- SottoASR
pretty_name: SottoASR转录清理数据集
configs:
- config_name: 默认
data_files:
- split: 训练集
path: data/train-*
- split: 验证集
path: data/validation-*
dataset_info:
features:
- name: input
dtype: 字符串
- name: output
dtype: 字符串
splits:
- name: train
num_bytes: 33192538
num_examples: 135503
- name: validation
num_bytes: 1296731
num_examples: 6921
download_size: 18979669
dataset_size: 34489269

---

# SottoASR转录清理数据集

<p align="center">
<a href="https://sotto.app">sotto.app</a> ·
<a href="https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m">训练模型（bf16）</a> ·
<a href="https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m-mlx-5bit">MLX 5比特模型</a>
</p>

## 概述

本数据集包含12.4万+条合成训练样本对，用于针对语音转文字转录清理任务微调小型大语言模型（Large Language Model，LLM）。本数据集曾用于训练[SottoASR转录清理模型](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m)——一款参数量为3.5亿的模型，在本任务上的表现优于带提示的20亿参数模型，且推理速度快8倍。

本数据集隶属于[**SottoASR**](https://sotto.app)——一款面向macOS的本地、隐私优先的自动语音识别（Automatic Speech Recognition，ASR）应用。

## 任务说明

**输入**：包含语音语病的原始小写无标点ASR转录文本
**输出**：移除语病、格式规范的干净文本

jsonl
{"input": "uh the server is uh running low on memory", "output": "The server is running low on memory."}
{"input": "use redis wait no memcached is better", "output": "Use Memcached."}
{"input": "ship it", "output": "Ship it."}
{"input": "send the email to john period", "output": "Send the email to John."}

## 类别

| 类别 | 占比 | 描述 |
|----------|---|-------------|
| 自我修正（self_correction） | 14% | 说话者在句中修正自身表述 |
| 保留原表述（preserve_wording） | 13% | 输入文本无需清理，模型不得过度编辑 |
| 填充词移除（filler_removal） | 11% | 移除uh、um、uhm、er、ah等填充词 |
| 混合类型（mixed） | 10% | 同时包含多种语病类型 |
| 赘语移除（crutch_words） | 8% | 移除basically、you know、I mean等赘语 |
| 未完成句移除（false_start） | 8% | 移除被放弃的句首内容 |
| 语音命令转换（dictation_commands） | 8% | 将“period”转换为“.”、“comma”转换为“,” |
| 误识别词修正（misheard_words） | 7% | 修正ASR识别错误（如将post gress修正为Postgres） |
| 语法修正（grammar） | 7% | 修正口语语法（如将gonna修正为going to） |
| 列表格式化（list_formatting） | 6% | 将口语化列表转换为编号格式 |
| 对抗性样本（adversarial） | 5% | 形似填充词但具有实际意义的词汇 |

## 应用领域

软件工程（24%）、通用商务（19%）、日常对话（15%）、医疗（10%）、法律（8%）、金融（7%）、技术（5%）、创意（5%）、学术（5%）

## 生成方法

采用三层构建方案：
1. **程序化注入（Layer 1）**：对公开的干净文本确定性地注入语音语病
2. **大语言模型生成（LLM-generated）**：通过Qwen3.5-35B和Grok 4.20生成符合上下文的样本
3. **人工撰写（Hand-crafted）**：由专家编写的边缘场景样本

本数据集的验证集通过率为94.6%。详细信息可参阅[训练研究文档](https://huggingface.co/juanquivilla/sotto-cleanup-lfm25-350m)。

## 数据划分

| 划分 | 样本数 |
|-------|---------|
| 训练集 | 118,069 |
| 验证集 | 6,215 |

## 许可证

MIT