Dataset-GAIA-Scaffold-Robustness

Description (HTML — paste into Zenodo's rich-text box) This dataset contains the raw .eval evaluation logs produced by Inspect AI for a single-researcher replication study of agent scaffold-effect findings on the GAIA benchmark (validation split, Levels 1 and 2). The study asks how much of a model's measured agentic capability is a property of the model versus the scaffold wrapped around it. Five models were each run through three agent scaffolds on the same GAIA questions, with three independent attempts per cell. Evaluation grid Models (5): claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-7, gemini-3.1-pro-preview, gpt-5.5 Scaffolds (3): S1 ReAct (baseline), S2 triframe (advisor/actor/rater), S3 planner-executor Attempts: 3 independent repeats per cell GAIA levels: Level 1 (53 questions) and Level 2 (86 questions), 2023_all validation split Tools: bash, python, web_search, text_editor, and a custom submit_answer tool Contents A directory tree of Inspect AI .eval log files organized by shard. Each shard directory is named {model}-{scaffold}-a{attempt}-L{level}-q{start}-{end} (e.g. opus-s2-a1-L2-q40-50). Every .eval file is a self-contained Inspect AI log containing the full model trajectory: messages, tool calls and results, token usage, timing, scores, and any errors. Files open with the inspect view log viewer or read programmatically via inspect_ai.log.read_eval_log. Format Inspect AI .eval format (a ZIP-based binary log container). Requires the inspect-ai Python package to parse. These are the complete raw logs; derived per-unit records, summary tables, and figures are produced from them by the accompanying analysis code. Data-quality notes The v1 run surfaced both provider-side billing events and provider-side SDK defects. Credit-exhaustion .eval files (containing only "credit balance is too low" API rejections, no model work) are quarantined separately and superseded by clean retries. Remaining errored samples carry classifiable signatures (provider_serialization_bug, prompt_too_long, content_filtering, pdf_page_limit, embedded_null_byte); these are documented and handled at analysis time rather than removed from the raw logs. Reuse Load these logs with Inspect AI to reproduce the accuracy, cost, and behavioral analyses, to inspect individual agent trajectories, or as a corpus of multi-scaffold agent runs on GAIA. The accompanying code repository documents the full experimental design, the three required Inspect/SDK patches, and the analysis pipeline. This dataset accompanies a pre-registered study; the full research questions, hypotheses, and implementation-defect policy live in the OSF pre-registration.