juliannunezb/smoltalk-gpt2-sft

---
language: en
license: apache-2.0
task_categories:
- text-generation
tags:
- conversational
- sft
- instruction-tuning
- gpt-2
- tokenized
- chat
- supervised-fine-tuning
size_categories:
- 1M<n<10M
configs:
- config_name: default
data_files:
- split: train
path: smoltalk_gpt2_sft_train.parquet
- split: val
path: smoltalk_gpt2_sft_val.parquet
---

# SmolTalk-GPT2-SFT

A **fully tokenized** version of the entire [`HuggingFaceTB/smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)
(`all` config) dataset — converted from raw multi-turn conversations into ready-to-train
**(token_id, loss_mask)** pairs using the **GPT-2 BPE tokenizer** (vocab=50257). Drop
it straight into a PyTorch trainer for **supervised fine-tuning (SFT)** of any
GPT-2-vocab language model — no template parsing, no role tagging, no chat-template
gymnastics required.

## Overview

- **Source**: [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) (`all` config) — 1.04M conversations curated by Hugging Face for SFT of small language models, mixing synthetic distillations from **Llama 3.1 405B Instruct** with public datasets covering math, code, summarization, function-calling, and conversation.
- **Tokenizer**: `tiktoken.get_encoding("gpt2")` — vocab 50257, EOT id **50256**.
- **Chat template**: deliberately minimal (no new special tokens added). Fits any model that uses GPT-2 BPE without extending the embedding table.
- **Loss mask**: per-token `uint8`. `1` on assistant response tokens (incl. trailing `<|endoftext|>`); `0` on the literal `User:` / `System:` / `Assistant:` prefixes and on user/system content.

## Dataset Statistics

| | train | val |
|---|---:|---:|
| conversations | 1,033,407 | 10,491 |
| total tokens | 951.4 M | 8.95 M |
| loss-active tokens | 673.0 M | 6.50 M |
| loss-active fraction | 70.7% | 72.6% |
| disk (zstd parquet) | 4.2 GB | 42 MB |

Train/val split: deterministic per-row dice roll (seed 42) at `VAL_FRAC = 0.01`.
**Skipped**: 19 source rows had no assistant message and were dropped.

## Mix Composition

The `all` config of smoltalk combines these subsets (counts are source-side; we kept all
that contained at least one assistant message):

| Subset | Source-side rows | Notes |
|---|---:|---|
| smol-magpie-ultra | 431,000 | Core Magpie distillation from Llama 3.1 405B Instruct |
| openhermes-100k | 100,000 | Subset of Teknium OpenHermes-2.5 |
| metamathqa-50k | 50,000 | Math word-problem reasoning |
| numina-cot-100k | 100,000 | Math chain-of-thought reasoning |
| apigen-80k | 80,000 | Function-calling traces |
| self-oss-instruct | ~50K | Code instructions (Starcoder2) |
| smol-summarize | 101,000 | Email + news summarization |
| smol-rewrite | 56,200 | Tone/style rewriting |
| smol-constraints | 36,200 | Format / constraint-following |
| explore-instruct-rewriting | 32,000 | Diverse text rewriting |
| systemchats-30k | 30,000 | System-prompt-style multi-turn |
| longalign | 3,730 | Long-context alignment |
| everyday-conversations | 2,380 | Casual everyday chat |

## Schema (Parquet)

Each row is one full conversation:

| column | type | meaning |
|---|---|---|
| `messages` | `list[struct{role: string, content: string}]` | Original conversation, bytes-identical to upstream |
| `text` | `string` | Rendered prompt — exactly the bytes that were tokenized |
| `token_ids` | `list[int32]` | GPT-2 BPE encoding of `text`. Contains real `<|endoftext|>` (50256) tokens at end of each assistant turn. |
| `loss_mask` | `list[int8]` | Same length as `token_ids`. `1` = train on this position, `0` = mask out of loss |

`text` is **byte-exact**: `tiktoken.encode(text, allowed_special={"<|endoftext|>"})` reproduces
`token_ids` exactly. So you can re-render with a different chat template by going back to
`messages` and re-tokenizing without losing fidelity.

## Chat Template

```
User: <user content>\n
Assistant: <assistant content><|endoftext|>
User: <next user content>\n
Assistant: <next assistant content><|endoftext|>
...
```

For system messages:

```
System: <system content>\n
User: ...
```

Design choices:
- **Single tokenizer, no new specials.** `<|endoftext|>` (token 50256, already in GPT-2 BPE) doubles as EOS *and* turn separator. No `<|im_start|>`-style ChatML tokens are added — this means the dataset is usable with any model that already speaks GPT-2 BPE, no embedding-table surgery required.
- **`Assistant:` prefix is masked** (loss = 0). The model sees it as a structural cue, not something it has to "generate." Loss only flows on the response itself (and the trailing `<|endoftext|>`).
- **Leading-space-aware tokenization.** The response is encoded as `" " + content` so BPE produces the natural ` Four` (id 6675) token instead of [` `, `Four`] — this avoids spurious extra tokens at every turn boundary. The full conversation `text` is byte-perfect round-trip with `token_ids`.

## Loading

### Standard `datasets` (whole-conversation rows)

```python
from datasets import load_dataset

ds = load_dataset("juliannunezb/smoltalk-gpt2-sft", split="train")
ex = ds[0]
print(ex["text"][:300]) # rendered conversation
print(len(ex["token_ids"])) # number of tokens
print(sum(ex["loss_mask"])) # tokens to compute loss on
```

### PyTorch (one conversation, masked SFT loss)

```python
import torch
import torch.nn.functional as F

ids = torch.tensor(ex["token_ids"], dtype=torch.long)
mask = torch.tensor(ex["loss_mask"], dtype=torch.float32)
x = ids[:-1]
y = ids[1:]
m = mask[1:] # mask aligned with prediction targets

logits = model(x.unsqueeze(0)) # (1, T-1, V)
per_tok = F.cross_entropy(
logits.view(-1, logits.size(-1)), y, reduction="none"
)
loss = (per_tok * m).sum() / m.sum().clamp(min=1.0)
```

### Memory-mapped packed format

For trainers that prefer flat `np.memmap` arrays (one big stream of tokens with a
matching mask), use the included `pack_sft_to_bin.py` to produce four `.bin` files:

```bash
python3 pack_sft_to_bin.py
# → sft_train_tokens.bin (uint16) ~1.9 GB
# → sft_train_mask.bin (uint8) ~960 MB
# → sft_val_tokens.bin
# → sft_val_mask.bin
```

Each conversation already ends with `<|endoftext|>`, so concatenating them yields a
clean stream where boundaries are obvious. A trainer using random-window sampling
will occasionally draw a window spanning two short conversations — fine, even
helpful (the model learns to "reset" on EOT).

## Reproducing

```bash
HF_TOKEN=... python3 prepare_sft.py # smol-magpie-ultra (~410K, smaller)
HF_TOKEN=... CONFIG=all python3 prepare_sft.py # full smoltalk (this release)
HF_TOKEN=... UPLOAD_TO=user/repo PRIVATE=0 python3 prepare_sft.py # also push to HF
```

Build details:
- **Built**: 2026-05-02 UTC, on a single M1 Max (~9 min for 1M conversations)
- **Script**: [`prepare_sft.py`](https://huggingface.co/datasets/juliannunezb/smoltalk-gpt2-sft/blob/main/prepare_sft.py)
- **Packer**: [`pack_sft_to_bin.py`](https://huggingface.co/datasets/juliannunezb/smoltalk-gpt2-sft/blob/main/pack_sft_to_bin.py)
- **Source dataset commit**: latest `HuggingFaceTB/smoltalk` (`all` config) at build time

### Sanity checks performed

- Round-trip `tiktoken.encode(text) == token_ids` verified on random samples spanning all major subsets (math, code, summarization, casual chat, system prompts).
- Loss-active fraction: 70.7% globally; varies per-subset (e.g. ~28% on long-prompt summarization, ~91% on short-prompt assistant-heavy turns).
- All `token_ids` validated `0 ≤ id < 50257`.
- All `loss_mask` values are `{0, 1}`.

## License

This dataset (rendered text, token_ids, loss_mask): **Apache-2.0**.

The `messages` column is a verbatim copy of the upstream `HuggingFaceTB/smoltalk`,
which is itself **Apache-2.0**. Underlying generations were synthesized with
**Llama 3.1 405B Instruct** by HuggingFaceTB via the Magpie pipeline — see the
upstream dataset card for full provenance and any subset-specific notes.

## Known Limitations

- **English-only**. The source dataset is English-centric.
- **Synthetic-heavy**. ~430K of the 1M conversations are Magpie distillations from a frontier model (Llama 3.1 405B). Inherits any biases/quirks of that model.
- **No safety filtering** beyond what HuggingFaceTB performed upstream.
- **GPT-2 BPE only**. If your target model uses a different tokenizer, you must re-tokenize from `messages` (the rendered `text` field has the data — but the `token_ids`/`loss_mask` columns will need rebuilding).
- **No system/multi-turn special tokens**. If you need ChatML or Llama-3 chat tokens for compatibility with downstream tooling, you'll want to render again.

## Citation

If you use this dataset, please cite the upstream SmolLM2 / SmolTalk paper:

```bibtex
@article{allal2025smollm2,
title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
author={Allal, Loubna Ben and Lozhkov, Anton and others},
journal={arXiv:2502.02737},
year={2025}
}
```

And, if you build on the tokenization/packaging work specifically:

```bibtex
@misc{smoltalk_gpt2_sft_2026,
title={SmolTalk-GPT2-SFT: tokenized smoltalk for GPT-2-vocab models},
author={Julián Núñez},
year={2026},
publisher={Hugging Face},
url={https://huggingface.co/datasets/juliannunezb/smoltalk-gpt2-sft}
}
```