The Tokenization Tax: Token Fertility and Production Impact Data for Vietnamese LLMs

Experimental dataset accompanying the paper 'The Tokenization Tax: Measuring Production Impact of Token Over-Consumption in Open-Source LLMs for Vietnamese-Language Workloads'. Contains raw token fertility measurements, production scenario latency logs, ROUGE and BERTScore results, and all experiment scripts for four open-source LLMs (Mistral 7B, Qwen 2.5 7B, SeaLLM 7B, VinaLLaMA 7B) on Vietnamese text across four benchmark corpora and three production workflow scenarios.