ether0-benchmark

[](https://huggingface.co/futurehouse/ether0)  # ether0-benchmark QA benchmark (test set) for the ether0 reasoning language model: https://huggingface.co/futurehouse/ether0 This benchmark is made from commonly used tasks - like reaction prediction in USPTO/ORD, molecular captioning from PubChem, or predicting GHS classification. It's unique from other benchmarks in that all answers are a molecule. It's balanced so that each task is about 25 questions, a reasonable amount for frontier model evaluations. The tasks generally follow previously reported numbers - e.g., a reaction prediction accuracy of 80% here would be approximately equivalent to performance on a withheld split of the USPTO-50k dataset. See our preprint [here](https://arxiv.org/abs/2506.17238) for more details on dataset construction and reward functions. The tasks in this test set include: - Completing SMILES fragments - Designing molecules adhering to molecular formula and functional group constraints - Predicting reaction outcomes - Proposing one-step synthesis pathways - Editing the solubility of a molecule - Converting IUPAC name to SMILES - Answering multiple-choice questions about safety, ADME properties, BBB permeability, toxicity, scent, and pKa We have measured the performance of ether0 and several frontier LLMs (at time of writing) on this benchmark:  ## Licensing This dataset repository is CC BY 4.0, copyright 2025 FutureHouse. ## Code Reward functions and problem templates used for this dataset can be found at https://github.com/Future-House/ether0.[](https://huggingface.co/futurehouse/ether0)  # ether0基准测试集 专为ether0推理型大语言模型打造的问答基准测试集（测试集）：https://huggingface.co/futurehouse/ether0 该基准测试集由多项通用任务构建而成，涵盖USPTO/ORD数据库中的反应预测、PubChem的分子标题生成，以及GHS（全球化学品统一分类和标签制度）分类预测等任务。本数据集与其他基准测试集的独特区别在于，所有问题的答案均为分子结构。数据集经过均衡设计，每项任务约包含25个问题，这一规模对于前沿大语言模型的评估而言十分合理。 各项任务的性能基准基本沿用已发表的参考数值——例如，本基准中反应预测准确率达到80%，大致等同于在USPTO-50k数据集的预留划分集上的表现。 如需了解数据集构建与奖励函数的更多细节，请查阅我们的预印本[此处](https://arxiv.org/abs/2506.17238)。本测试集包含以下任务： - 补全SMILES（简化分子线性输入规范）片段 - 设计符合分子式与官能团约束条件的分子 - 预测化学反应产物 - 提出单步合成路径 - 修改分子的溶解度属性 - 将IUPAC（国际纯粹与应用化学联合会）命名转换为SMILES格式 - 回答关于安全性、ADME（吸收、分布、代谢、排泄）性质、BBB（血脑屏障）通透性、毒性、气味以及pKa（酸解离常数）的多项选择题 我们已在该基准测试集上评估了ether0以及截至撰写本文时的多款前沿大语言模型的性能：  # 许可协议 本数据集仓库采用CC BY 4.0许可协议，版权归2025 FutureHouse所有。 # 代码 本数据集所使用的奖励函数与问题模板可从https://github.com/Future-House/ether0获取。

搜集汇总

背景与挑战

背景概述

该数据集是专为ether0推理语言模型设计的QA基准测试集，所有答案均为分子，涵盖反应预测、分子设计等常见任务，每个任务包含约25个问题，适用于前沿模型评估。数据集遵循CC BY 4.0许可，并提供相关代码和详细信息。