manassehzw/sna-dataset

---
dataset_info:
features:
- name: audio
dtype: audio
- name: transcription
dtype: string
- name: source_id
dtype: string
- name: source_speaker_id
dtype: string
- name: speaker_idx
dtype: int64
- name: speaker_clip_count
dtype: int64
- name: language
dtype: string
- name: gender
dtype: string
- name: has_punctuation
dtype: bool
- name: snr_db
dtype: float64
- name: speech_ratio
dtype: float64
- name: quality_score
dtype: float64
- name: duration
dtype: float64
splits:
- name: train
num_examples: 13532
- name: validation
num_examples: 1640
- name: test
num_examples: 1808
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
- split: test
path: data/test-*
license: cc-by-4.0
task_categories:
- automatic-speech-recognition
- text-to-speech
language:
- sna
tags:
- audio
- speech
- shona
- asr
- tts
- african-language
source_datasets:
- google/WaxalNLP
pretty_name: Shona Speech Dataset (SNA)
size_categories:
- 10K<n<100K
---

# Shona Speech Dataset (sna-dataset)

A cleaned, metadata-rich Shona (`sna`) speech dataset prepared through a reproducible data engineering pipeline for downstream **ASR** and **TTS** workflows.

This release is intended as a **general-purpose standard corpus**: quality metadata is provided, but aggressive opinionated filtering is avoided so users can apply task-specific thresholds.

## Dataset Details

### Dataset Description

- **Curated by:** [Manasseh Changachirere (Harare Institute of Technology)](https://www.manasseh.dev/)
- **Derived from:** [google/WaxalNLP](https://huggingface.co/datasets/google/WaxalNLP/)
- **Language:** Shona 🇿🇼
- **Repository:** `manassehzw/sna-dataset`
- **Total Clips:** 16,980
- **Total Speech Hours:** 86.222 hours
- **Unique Speakers:** 133

### Dataset Sources

- **Source dataset:** [google/WaxalNLP](https://huggingface.co/datasets/google/WaxalNLP)
- **Subset used:** `sna_asr` (train + validation + test labeled splits)
- **Pipeline repository:** `sna-data-pipeline` (Modal-based end-to-end processing)

## Annotation Disclaimer

Some metadata fields (especially `source_speaker_id` and `gender`) are inherited from the original source dataset may contain labeling errors.
This release preserves source-provided identity annotations for provenance and reproducibility. While quality-control and audit checks were applied, metadata assignments are **not guaranteed to be 100% correct**.
Future revisions may include improved speaker-consistency and sex/gender label audits, with any corrections documented in the changelog.

## Uses

### Direct Use

- Automatic Speech Recognition (ASR) training and fine-tuning
- Speech-text data analysis in Shona
- TTS subset construction via metadata filters (e.g., speaker and quality constraints)

### Out-of-Scope Use

- Clinical, forensic, or speaker verification use without additional validation
- Claims of broad dialect/sociolinguistic representativeness without dedicated bias study

## Dataset Structure

- `train`: 13,532 clips
- `validation`: 1,640 clips
- `test`: 1,808 clips

The split is **speaker-stratified by clip proportion** using `speaker_idx` (not speaker-disjoint), so each split retains similar speaker distribution while preserving clip diversity.

## Data Fields

- **`audio`**: 24kHz mono float audio
- **`transcription`**: normalized Shona transcription
- **`source_id`**: original clip identifier from source dataset
- **`source_speaker_id`**: original speaker hash from source dataset
- **`speaker_idx`**: stable integer speaker index (frequency-sorted at ingest)
- **`speaker_clip_count`**: number of clips for that speaker in the cleaned dataset
- **`language`**: normalized language code (`sna`)
- **`gender`**: normalized gender label (`Male` / `Female`)
- **`has_punctuation`**: whether transcript contains sentence punctuation
- **`snr_db`**: signal-to-noise proxy metric
- **`speech_ratio`**: fraction of VAD frames classified as speech
- **`quality_score`**: composite metric (`snr_db` with reliability penalties)
- **`duration`**: post-processing clip duration in seconds

## Dataset Creation

### Curation Rationale

The goal was to produce a robust, well-documented Shona speech corpus that preserves provenance, adds high-value metadata, and remains broadly usable for both ASR and TTS downstream tasks.

### Data Processing Pipeline

## Source Code

The dataset was produced with a reproducible, Modal-based data engineering pipeline implemented in Python (ingest -> metadata annotation -> text normalization -> audio normalization -> cleanup -> split/upload).
Pipeline source code: [sna-data-pipeline](https://github.com/manasseh-zw/sna-data-pipeline)

1. **Ingest**
- Loaded labeled `sna_asr` splits from WaxalNLP
- Preserved provenance by renaming:
- `id` → `source_id`
- `speaker_id` → `source_speaker_id`
- Assigned stable `speaker_idx`

2. **Metadata Annotation**
- Normalized `language` and `gender`
- Added `speaker_clip_count`

3. **Text Normalization**
- Standardized punctuation/quotes/dashes
- Cleaned unsupported symbols
- Added `has_punctuation`

4. **Audio Normalization**
- Resampled to 24kHz mono
- WebRTC VAD trimming with smoothing
- Internal silence-gap compression
- Computed `snr_db`, `speech_ratio`, `quality_score`, `duration`
- Dropped zero-speech/empty clips and blacklisted speaker clips

5. **Post-normalization Cleanup**
- Dropped very short clips (`duration` threshold)
- Dropped singleton-speaker rows
- Refreshed `speaker_clip_count`

6. **Split and Upload**
- Speaker-stratified 80/10/10 split
- Published as `sna-dataset` with dataset card + audit trail

## Bias, Risks, and Limitations

- Inherits demographic/dialectal distribution from source data.
- Not speaker-disjoint across splits (same speaker can appear in multiple splits).
- Quality metrics are useful for ranking/filtering, but not absolute perceptual guarantees.
- Users should apply task-specific filtering for strict TTS quality targets.

## Citation

Prepared by Manasseh Changachirere (Harare Institute of Technology).

If you use this dataset, please cite the original source:

```bibtex
@inproceedings{niang2024waxalnlp,
title={WaxalNLP: A Large Scale High Quality Speech Dataset for African Languages},
author={Niang, El Hadj Mamadou and Dieng, Moustapha and Ba, Thierno Ibrahima and Ndiaye, Mamadou Boumedine and others},
booktitle={Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
year={2024}
}