TTS-AGI/voice-taxonomy-flash-train

---
license: cc-by-4.0
task_categories:
- audio-classification
tags:
- voice
- speech
- taxonomy
- whisper
- gemini
- tts
- voice-attributes
size_categories:
- 10K<n<100K
---

# Voice Taxonomy Fine-tuning Dataset (Gemini Flash)

**36,641 speech samples** annotated with **57 voice taxonomy dimensions** (0-6 ordinal scale) by **Gemini Flash**. Carefully balanced (~100 samples per bucket per dimension) for fine-tuning voice attribute classifiers.

## Related Datasets

| Dataset | Purpose | Link |
|---------|---------|------|
| Pre-training (large, Whisper ensemble) | Pre-training | [TTS-AGI/voice-taxonomy-pretrain](https://huggingface.co/datasets/TTS-AGI/voice-taxonomy-pretrain) |
| **This dataset** | Fine-tuning (balanced, high-quality) | — |
| Validation (Gemini 3.1 Pro gold) | Evaluation | [TTS-AGI/voice-taxonomy-val](https://huggingface.co/datasets/TTS-AGI/voice-taxonomy-val) |

## Format

WebDataset TAR with MP3+JSON pairs:
```
{stem}.mp3 # Audio (mono, 44.1kHz, 64kbps, ≤30s)
{stem}.json # 57-dim taxonomy annotation
```

Each JSON:
```json
{
"AGEV": {"value": 3, "name": "Perceived Age", "label": "young adult"},
"GEND": {"value": 5, "name": "Gender Presentation", "label": "standard masculine"},
"TEMP": {"value": 4, "name": "Tempo", "label": "slightly fast energetic"},
...
}
```

## Training Plan

See [TRAINING_PLAN.md](TRAINING_PLAN.md) for the full training strategy and `train_voice_taxonomy.py` for a self-contained training script.

## Quick Start

```bash
# Download all 3 datasets
huggingface-cli download TTS-AGI/voice-taxonomy-pretrain --local-dir pretrain
huggingface-cli download TTS-AGI/voice-taxonomy-flash-train --local-dir finetune
huggingface-cli download TTS-AGI/voice-taxonomy-val --local-dir val

# Fine-tune (after pre-training)
python train_voice_taxonomy.py --phase finetune --encoder laion/BUD-E-Whisper --gpu 0 \
--resume checkpoints/pretrain_best.pt \
--finetune-tar finetune/voice_taxonomy_flash_train.tar \
--val-tar val/voice_taxonomy_val.tar
```

## Balancing Strategy

Samples were selected to maximize coverage across all 57 × 7 = 399 buckets:
- Up to 100 samples per bucket per dimension
- Files deduplicated across dimensions
- Validation set files excluded
- Total: 36,641 unique files from 318K candidates

## Labels

Labels were generated by **Gemini 2.0 Flash** via multimodal audio annotation with a detailed system prompt covering all 57 dimensions. Anti-center-bias instructions ensure good distribution across the 0-6 scale.

## Taxonomy

57 dimensions covering: speaker identity, timbral quality, resonance placement, prosody, articulation, emotion, and speaking style. See `taxonomy_labels.json` for full definitions.

---
许可协议：CC BY 4.0
任务类别：
- 音频分类（audio-classification）
标签：
- 语音（voice）
- 言语（speech）
- 分类体系（taxonomy）
- Whisper
- Gemini
- 文本到语音合成（Text-to-Speech, TTS）
- 语音属性（voice-attributes）
样本规模区间：
- 10K<n<100K
---

# 语音分类体系微调数据集（Gemini Flash）

**36641条语音样本**由**Gemini Flash**基于0至6的序数评分尺度，对57项语音分类体系维度完成标注。为适配语音属性分类器的微调需求，数据集经过严格均衡处理（每个维度的每个分桶约含100条样本）。

## 相关数据集

| 数据集 | 用途 | 链接 |
|---------|---------|------|
| 预训练数据集（大规模，Whisper集成模型） | 预训练 | [TTS-AGI/voice-taxonomy-pretrain](https://huggingface.co/datasets/TTS-AGI/voice-taxonomy-pretrain) |
| **本数据集** | 微调（均衡且高质量） | — |
| 验证数据集（Gemini 3.1 Pro gold） | 模型评估 | [TTS-AGI/voice-taxonomy-val](https://huggingface.co/datasets/TTS-AGI/voice-taxonomy-val) |

## 数据格式

采用Web数据集（WebDataset）TAR格式，包含MP3音频与JSON标注配对文件：

{stem}.mp3 # 音频（单声道，44.1kHz采样率，64kbps码率，时长≤30秒）
{stem}.json # 57维度分类体系标注文件

单个JSON文件格式如下：
json
{
"AGEV": {"value": 3, "name": "感知年龄", "label": "青年成年"},
"GEND": {"value": 5, "name": "性别呈现", "label": "标准男性化"},
"TEMP": {"value": 4, "name": "语速", "label": "稍快且充满活力"},
...
}

## 训练方案

完整训练策略详见[TRAINING_PLAN.md](TRAINING_PLAN.md)，独立训练脚本可参考`train_voice_taxonomy.py`。

## 快速上手

bash
# 下载全部3个数据集
huggingface-cli download TTS-AGI/voice-taxonomy-pretrain --local-dir pretrain
huggingface-cli download TTS-AGI/voice-taxonomy-flash-train --local-dir finetune
huggingface-cli download TTS-AGI/voice-taxonomy-val --local-dir val

# 预训练完成后执行微调
python train_voice_taxonomy.py --phase finetune --encoder laion/BUD-E-Whisper --gpu 0
--resume checkpoints/pretrain_best.pt
--finetune-tar finetune/voice_taxonomy_flash_train.tar
--val-tar val/voice_taxonomy_val.tar

## 均衡策略

样本筛选旨在最大化覆盖57×7=399个分桶：
- 每个维度的每个分桶最多保留100条样本
- 跨维度的文件已完成去重
- 已排除验证集文件
- 最终从31.8万候选样本中筛选出36641条唯一文件

## 标注说明

标注由**Gemini 2.0 Flash**通过多模态音频标注生成，配套涵盖全部57个维度的详细系统提示。同时加入反中心偏差指令，确保评分在0-6尺度上分布均匀。

## 分类体系

57个维度涵盖说话人身份、音色质量、共鸣位置、韵律、发音清晰度、情感与说话风格。完整维度定义详见`taxonomy_labels.json`文件。