Flakiness in Autonomous Test Agents - Dataset

This dataset accompanies the paper "Flakiness in Autonomous Test Agents" and provides all artifacts generated during our empirical study on flakiness in LLM-based Autonomous Test Agents (ATAs).

Contents

The dataset includes:

ATA Execution Traces: Complete execution traces from the Autonomous Test Agent running a benchmark of 113 professionally authored end-to-end web tests across three web applications (Classifieds, Postmill, and OneStopMarket). The traces capture:

Full agent-LLM conversation histories

Actions performed and their outcomes

Assertion evaluations and verdicts

Multiple runs per test case under identical conditions (same model, prompts, temperature, and environment)

Knowledge Extraction Agent Conversations: Complete LLM conversation logs from the Knowledge Extraction Agent, which analyzes execution traces to generate Knowledge Items. These conversations document:

Trace analysis and identification of instability patterns

Knowledge Item proposals (52 total)

Alternative formulations provided for human validation

Context

The ATA uses Anthropic Claude Haiku as its underlying LLM, configured with a sampling temperature of 0.2. Each test was executed three times to identify flaky behavior, defined as inconsistent outcomes or trajectories under identical run conditions.

Purpose

This dataset enables researchers to:

Analyze ATA behavior and decision-making through detailed execution traces

Study the causes and patterns of flakiness in LLM-based testing agents

Examine the Knowledge Item extraction process and its effectiveness

Replicate or extend our flakiness measurement methodology

Develop improved stabilization techniques for autonomous test agents