Autonomous Tester Agent Benchmark

This is a benchmark for Autonomous Tester Agents. This benchmark is composed of a hundred test cases for 3 web applications written by professional testers. Half of the test cases are mutations of the original ones to better evaluate the capacity of agents to identify bugs. The test cases are composed of several steps that need precise assertion.

An evaluation of the SeeAct model can also be found in the repository, as well as the source code to replicate the results. This repository is the replication package of a paper currently submitted to the ISSTA conference. (https://conf.researchr.org/home/issta-2025)

本数据集为面向自主测试智能体（Autonomous Tester Agents）的基准测试集。该基准测试集由专业测试人员针对3款Web应用编写的100条测试用例构成，其中半数测试用例为原始用例的变异版本，用于更精准地评估智能体识别软件缺陷的能力。所有测试用例均包含若干需要执行精确断言的步骤。
本仓库中还包含SeeAct模型的评估方案，以及用于复现实验结果的源代码。本仓库为一篇已投稿至ISSTA会议的论文的复现包，该会议官网为https://conf.researchr.org/home/issta-2025。