idobrovolskyi/cyrillic-vs-latin-tokenization

Cyrillic vs Latin Tokenization Benchmark是一个跨语言基准数据集，用于测量5种语言（乌克兰语、塞尔维亚语、哈萨克语、阿塞拜疆语、乌兹别克语）和6种现代大型语言模型tokenizer（如o200k_base、Llama 4 Scout、Gemma 4、Qwen 3.5、Mistral Small、Grok）中Cyrillic与Latin脚本的tokenization成本。数据集基于这些语言的词典资源构建，包含单词形式，旨在研究tokenizer对脚本的偏好是否与每种语言在网络上的主导脚本一致。关键发现表明，tokenizer的偏好总是匹配语言的主导网络脚本，脚本税并非Cyrillic固有，而是由tokenizer训练数据决定。数据集还包括罗马化标准、数据来源和统计分析，用于支持跨语言NLP研究。

Cyrillic vs Latin Tokenization Benchmark is a cross-lingual benchmark measuring the tokenization cost of Cyrillic vs Latin script across 5 languages (Ukrainian, Serbian, Kazakh, Azerbaijani, Uzbek) and 6 modern LLM tokenizers (e.g., o200k_base, Llama 4 Scout, Gemma 4, Qwen 3.5, Mistral Small, Grok). The dataset is built from dictionary resources for these languages, containing word forms, and aims to investigate whether tokenizer preferences align with the dominant web script for each language. Key findings show that tokenizer preference always matches the dominant web script, and the script tax is not inherent to Cyrillic but determined by tokenizer training data. It includes romanization standards, data sources, and statistical analysis to support cross-lingual NLP research.