foundry-ai/swe-prbench

---
license: cc-by-4.0
task_categories:
- text-classification
language:
- en
tags:
- code-review
- benchmark
- llm-evaluation
- pull-requests
- software-engineering
pretty_name: SWE-PRBench
size_categories:
- n<1K
configs:
- config_name: prs
data_files: dataset/prs.jsonl
- config_name: eval_split
data_files: dataset/evals/eval_100.json
---

# SWE-PRBench

**Benchmarking AI Code Review Quality Against Human Pull Request Feedback**

- 📖 Blog: [Read the blog](https://foundryhq.ai/blog/swe-prbench-benchmarking-ai-code-review-quality)
- 💻 GitHub Repository: [View the code](https://github.com/FoundryHQ-AI/swe-prbench)

---

## Overview

SWE-PRBench is a benchmark of **350 pull requests with human-annotated
ground truth** for evaluating whether LLMs can identify the same issues
that real human reviewers flag in production code.

Existing benchmarks like SWE-Bench measure whether models can *produce*
correct code. SWE-PRBench measures whether a model can *evaluate* proposed
code changes as an expert reviewer would — a fundamentally different
judgment task with no pass/fail test suite and no single correct answer.

**Key result:** 8 frontier models detect only 15–31% of human-flagged
issues on the diff-only configuration. All 8 models degrade monotonically
as context expands, establishing that attention representation — not content
selection — is the binding constraint for AI code review.

---

## Why SWE-PRBench?

Existing code review datasets and tools fall short in three ways: they use
synthetic or generated ground truth, they do not structure evaluation around
issue detection capability, and none provide controlled context configurations
for ablation. SWE-PRBench addresses all three gaps.

| Property | CodeReviewer | DeepCRCEval | RovoDev | SWE-PRBench (Ours) |
|----------|-------------|-------------|---------|-------------------|
| Primary contribution | Model + dataset | Eval metrics | Production tool | Dataset + protocol |
| Ground truth source | Synthetic pairs | Generated | CRR metric only | **Human reviewers** |
| Source links retained | No | No | N/A | **Yes** |
| Difficulty taxonomy | None | None | None | **3 types** |
| Context configurations | None | None | None | **3 frozen** |
| Issue detection eval | No | No | Partial | **Yes** |
| Judge validated | No | No | No | **κ=0.75** |
| Public dataset | Partial | No | No | **Yes** |

Ground truth in SWE-PRBench consists of review comments written by human
engineers during the actual review process on real merged pull requests,
collected after the fact via GitHub's review API. No comments are generated,
synthesised, or modified during dataset construction.

---

## Leaderboard (Paper Baseline)


| Rank | Model | Overall (s̄) | DR_A | FPR |
|------|-------|-------------|------|-----|
| 1 | Claude Haiku 4.5 | 0.153 | 0.306 | 0.346 |
| 2 | Claude Sonnet 4.6 | 0.152 | 0.297 | 0.227 |
| 3 | DeepSeek V3 | 0.150 | 0.312 | 0.315 |
| 4 | Mistral Large 3 | 0.147 | 0.305 | 0.353 |
| 5 | GPT-4o | 0.113 | 0.220 | 0.193 |
| 6 | GPT-4o-mini | 0.108 | 0.210 | 0.353 |
| 7 | Mistral Small | 0.106 | 0.257 | 0.251 |
| 8 | Llama 3.3 70B | 0.079 | 0.223 | 0.417 |

Evaluated on `evals/eval_100.json`. Judge: GPT-5.2. Pipeline: v0.4.1.

---

## Dataset at a Glance

| Property | Value |
|----------|-------|
| Total PRs | 350 |
| Repositories | 65 across 100 RQS-qualified repos |
| Languages | Python (69%), JS (11%), Go (10%), TypeScript (6%), Java (4%) |
| Difficulty types | Type1\_Direct / Type2\_Contextual / Type3\_Latent |
| Context configs | config\_A (2k) / config\_B (2.2k) / config\_C (2.5k) tokens |
| Ground truth | Real human reviewer comments, not generated or synthesised |
| Judge validation | κ=0.75 (substantial agreement) |
| Pipeline version | v0.4.1 |

---

## Dataset Preparation

Dataset construction follows a four-stage pipeline designed to ensure
ground truth quality, repository diversity, and contamination resistance.

**Stage 1 — Repository selection via RQS.**
Repositories are scored using a Repository Quality Score (RQS) across five
dimensions: review culture (share of substantive human comments), PR recency,
test quality, PR volume, and contamination risk (inverse star count). Only
repositories scoring ≥60/100 are included, ensuring that ground truth comes
from codebases with genuine human review activity.

**Stage 2 — PR collection and filtering.**
For each qualifying repository, merged pull requests are collected via
GitHub's GraphQL and REST APIs over a six-month window. PRs pass through
a ten-stage hard filter covering: merged-only status, minimum two substantive
human comments, non-documentation changes, no automated dependency updates
(Dependabot/Renovate), and explicit AI comment detection. PRs where more
than 30% of review comments originate from known AI bots or match
AI-generated structural patterns are excluded, preserving the integrity of
human ground truth.

**Stage 3 — PR quality scoring via RVS.**
Each PR is scored using a PR Review Value Score (RVS) combining review
depth, code complexity, discussion signal, test change signal, and bug-fix
signal. Only PRs with RVS ≥ 0.35 enter the final dataset, ensuring every
benchmark task carries meaningful ground-truth signal.

**Stage 4 — Difficulty classification.**
Each PR is classified into one of three difficulty types based on where
the evidence for a reviewable issue resides: directly in the diff
(Type1\_Direct), in surrounding unchanged code (Type2\_Contextual), or in
dependent files (Type3\_Latent). Classification is derived automatically
from the `is_in_diff` field of human reviewer comments cross-referenced
against diff hunk line ranges.

The result is 350 PRs from 65 repositories across 6 languages, with a
construction funnel of ~3,000 raw PRs → 700 after hard filtering → 350
after RVS quality cut.

---

## Difficulty Taxonomy

Each PR is classified by where the evidence for a reviewable issue resides:

- **Type1\_Direct (66%)** — Issue is directly visible in the changed lines. A reviewer needs only the diff to identify it.
- **Type2\_Contextual (21%)** — Issue requires understanding changed code relative to surrounding unchanged code in the same file.
- **Type3\_Latent (12%)** — Issue resides in files that import or depend on the changed files. Requires cross-file reasoning.

---

## Context Configurations

Three frozen configurations enable systematic ablation of context provision:

| Config | Layers | Real-world analogue | Token budget |
|--------|--------|---------------------|-------------|
| config\_A | Task focus, summary, diff, metadata | GitHub PR email notification | 2,000 |
| config\_B | + Execution context, behaviour mapping | GitHub PR web view | 2,200 |
| config\_C | + Test signatures | Reviewer with full IDE access | 2,500 |

Configs differ in **layer composition**, not token volume. The A>B>C
degradation implicates attention representation, not context length.
Pre-built contexts for all 350 PRs are released as frozen artefacts at
pipeline version v0.4.1.

---

## Dataset Structure
```
dataset/
├── prs.jsonl # 350 PR records (metadata + diffs)
├── annotations/ # 350 human annotation files (ground truth)
│ ├── dask__12221_human.json
│ └── ...
├── contexts/
│ ├── config_A/ # 350 pre-built config_A contexts
│ ├── config_B/ # 350 pre-built config_B contexts
│ └── config_C/ # 350 pre-built config_C contexts
└── evals/
└── eval_100.json # 100-PR stratified sample used in paper
```

---

## File Formats

**`prs.jsonl`** — one line per PR:
```json
{
"task_id": "dask__12221",
"repo": "dask/dask",
"language": "Python",
"difficulty": "Type1_Direct",
"rvs_score": 0.52,
"diff_patch": "diff --git ...",
"base_commit": "0a075534...",
"head_commit": "59dab320...",
"num_substantive_comments": 3
}
```

**`annotations/dask__12221_human.json`** — ground truth:
```json
{
"task_id": "dask__12221",
"comments": [
{
"comment_id": "c_1",
"body": "Out of scope: this should belong to os.process_cpu_count...",
"file": "dask/system.py",
"line": 82,
"is_in_diff": true,
"is_initiating_comment": true
}
]
}
```

**`contexts/config_A/dask__12221.json`**:
```json
{
"task_id": "dask__12221",
"config_name": "config_A",
"pipeline_version": "v0.4.1",
"total_tokens": 847,
"was_truncated": false,
"rendered": "## Layer 0 - Task + Focus\n..."
}
```

**`evals/eval_100.json`** — paper evaluation split:
```json
{
"description": "100-PR stratified sample used in paper baseline.",
"n": 100,
"stratification": {
"Type1_Direct": 40,
"Type2_Contextual": 40,
"Type3_Latent": 20
},
"task_ids": ["dask__12221", "prowler__9865", "..."]
}
```

---

## Ground Truth

Ground truth consists of review comments written by human engineers
during the actual review process on real merged pull requests. Comments
are collected from GitHub's review API after the fact. **No comments are
generated, synthesised, or modified during dataset construction.**

Ground truth inclusion criteria (from `RUBRIC.md`):
- Human-authored
- Initiating comment, not a reply
- ≥10 words
- References specific code behaviour
- Not pure praise

---

## Leaderboard Submission

All 350 PRs are publicly released following the honor-system standard
established by SWE-Bench and SWE-Bench Pro. Evaluation harness and
submission instructions coming shortly. To register interest in
submitting, open a discussion on this dataset page.

---

## Citation
```bibtex
@article{kumar2026sweprbench,
title={SWE-PRBench: Benchmarking AI Code Review Quality
Against Real Pull Request Feedback},
author={Kumar, Deepak},
journal={arXiv preprint},
year={2026}
}
```

---

## License

Dataset: CC BY 4.0

---
许可证：CC BY 4.0
任务类别：
- 文本分类
语言：
- 英语
标签：
- 代码审查
- 基准测试
- 大语言模型（LLM）评估
- 拉取请求（Pull Requests）
- 软件工程
美观名称：SWE-PRBench
规模类别：
- 少于1000条
配置项：
- 配置名称：prs
数据文件：dataset/prs.jsonl
- 配置名称：eval_split
数据文件：dataset/evals/eval_100.json
---

# SWE-PRBench

**对标人类拉取请求反馈的AI代码审查质量基准测试**

- 📖 博客：[阅读博客](https://foundryhq.ai/blog/swe-prbench-benchmarking-ai-code-review-quality)
- 💻 GitHub 仓库：[查看代码](https://github.com/FoundryHQ-AI/swe-prbench)

## 概述

SWE-PRBench是一个包含350条带人类标注真实标签（ground truth）的基准数据集，用于评估大语言模型（LLM）能否识别出人类专业评审在生产代码中标记的同类问题。

现有基准如SWE-Bench旨在评估模型能否生成正确代码，而SWE-PRBench则用于评估模型能否像专业评审一样评估提交的代码变更——这是一项本质不同的评判任务，既无通过/不通过的测试套件，也不存在唯一正确答案。

**核心结果**：8款前沿模型在仅包含代码差异（diff）的配置下，仅能识别出15%至31%的人类标记问题。且所有8款模型的性能随上下文扩展呈单调下降趋势，这证明注意力表征（而非内容选择）是制约AI代码审查效果的核心瓶颈。

## 为何选择SWE-PRBench？

现有代码审查数据集与工具存在三大短板：其一，采用合成或生成的真实标签；其二，未围绕问题检测能力构建评估框架；其三，均未提供可用于消融实验的可控上下文配置。SWE-PRBench则填补了上述所有空白。

| 属性 | CodeReviewer | DeepCRCEval | RovoDev | SWE-PRBench（本工作） |
|----------|-------------|-------------|---------|-------------------|
| 核心贡献 | 模型+数据集 | 评估指标 | 生产工具 | 数据集+评估协议 |
| 真实标签来源 | 合成样本对 | 生成内容 | 仅CRR指标 | **人类评审员** |
| 保留源码链接 | 否 | 否 | 不适用 | **是** |
| 难度分类体系 | 无 | 无 | 无 | **3种类型** |
| 上下文配置 | 无 | 无 | 无 | **3种固定配置** |
| 问题检测评估 | 否 | 否 | 部分支持 | **是** |
| 评审验证 | 否 | 否 | 否 | **κ=0.75** |
| 公开数据集 | 部分公开 | 不公开 | 不公开 | **是** |

SWE-PRBench的真实标签均为人类工程师在实际审查流程中，针对真实合并拉取请求所撰写的审查评论，通过GitHub审查API事后收集得到。在数据集构建过程中，未生成、合成或修改任何评论。

## 基准榜单（论文基线）


| 排名 | 模型 | 整体得分（s̄） | DR_A | FPR |
|------|-------|-------------|------|-----|
| 1 | Claude Haiku 4.5 | 0.153 | 0.306 | 0.346 |
| 2 | Claude Sonnet 4.6 | 0.152 | 0.297 | 0.227 |
| 3 | DeepSeek V3 | 0.150 | 0.312 | 0.315 |
| 4 | Mistral Large 3 | 0.147 | 0.305 | 0.353 |
| 5 | GPT-4o | 0.113 | 0.220 | 0.193 |
| 6 | GPT-4o-mini | 0.108 | 0.210 | 0.353 |
| 7 | Mistral Small | 0.106 | 0.257 | 0.251 |
| 8 | Llama 3.3 70B | 0.079 | 0.223 | 0.417 |

评估基于`evals/eval_100.json`，评审模型为GPT-5.2，评估流水线版本为v0.4.1。

## 数据集概览

| 属性 | 数值 |
|----------|-------|
| 拉取请求总数 | 350 |
| 仓库数量 | 65个（来自100个符合仓库质量评分（Repository Quality Score, RQS）标准的仓库） |
| 编程语言占比 | Python（69%）、JavaScript（JS，11%）、Go（10%）、TypeScript（6%）、Java（4%） |
| 难度类型 | 直接型（Type1_Direct）/上下文型（Type2_Contextual）/隐式型（Type3_Latent） |
| 上下文配置 | config_A（2000 Token（Token））/ config_B（2200 Token（Token））/ config_C（2500 Token（Token）） |
| 真实标签 | 真实人类评审评论，非生成或合成内容 |
| 评审一致性 | κ=0.75（一致性较强） |
| 流水线版本 | v0.4.1 |

## 数据集构建流程

数据集构建采用四阶段流水线，旨在保障真实标签质量、仓库多样性与数据污染抗性。

**阶段1：基于仓库质量评分（RQS）的仓库筛选**
仓库通过RQS从五个维度进行打分：审查文化（人类有效评论占比）、拉取请求更新频率、测试质量、拉取请求数量与污染风险（星标数的倒数）。仅纳入得分≥60/100的仓库，确保真实标签源自具备真实人类审查活动的代码库。

**阶段2：拉取请求收集与筛选**
针对每个符合条件的仓库，通过GitHub的GraphQL与REST API在六个月的时间窗口内收集已合并的拉取请求。拉取请求需通过十重严格筛选，包括：仅保留已合并状态、至少两条人类有效评论、非文档类变更、无自动化依赖更新（Dependabot/Renovate）、以及明确的AI评论检测。若拉取请求中超过30%的审查评论来自已知AI机器人或匹配AI生成的结构模式，则将其排除，以保障人类真实标签的完整性。

**阶段3：基于拉取请求审查价值评分（PR Review Value Score, RVS）的质量打分**
每个拉取请求通过RVS进行打分，该评分综合了审查深度、代码复杂度、讨论热度、测试变更信号与缺陷修复信号。仅保留RVS≥0.35的拉取请求，确保每个基准任务都具备有意义的真实标签信号。

**阶段4：难度分类**
每个拉取请求根据可审查问题的证据位置被划分为三类难度：直接存在于代码差异中（Type1_Direct）、存在于同一文件中未变更的周边代码内（Type2_Contextual）、或存在于依赖文件中（Type3_Latent）。分类自动基于人类评审评论的`is_in_diff`字段，并结合代码差异块的行范围进行交叉验证。

最终数据集包含来自6种编程语言、65个仓库的350条拉取请求，其构建漏斗为：约3000条原始拉取请求 → 经过十重筛选后剩余700条 → 经RVS质量筛选后最终得到350条。

## 难度分类体系

每个拉取请求根据可审查问题的证据位置进行分类：

- **直接型（Type1_Direct，占比66%）**：问题直接出现在变更行中，评审仅需查看代码差异即可识别。
- **上下文型（Type2_Contextual，占比21%）**：问题需要结合同一文件中未变更的周边代码来理解变更代码。
- **隐式型（Type3_Latent，占比12%）**：问题存在于导入或依赖变更文件的其他文件中，需要跨文件推理。

## 上下文配置

三种固定配置支持对上下文提供方式的系统性消融实验：

| 配置 | 层级组成 | 现实类比 | Token预算 |
|--------|--------|---------------------|-------------|
| config_A | 任务聚焦、摘要、代码差异、元数据 | GitHub拉取请求邮件通知 | 2000 |
| config_B | 加上执行上下文、行为映射 | GitHub拉取请求网页视图 | 2200 |
| config_C | 加上测试签名 | 具备完整集成开发环境（Integrated Development Environment, IDE）访问权限的评审 | 2500 |

各配置的差异在于**层级组成**，而非Token数量。config_A到config_B到config_C的性能下降趋势表明，制约因素是注意力表征而非上下文长度。本工作已针对所有350条拉取请求预构建上下文，并以固定工件形式在流水线版本v0.4.1中发布。

## 数据集结构

dataset/
├── prs.jsonl # 350条拉取请求记录（元数据+代码差异）
├── annotations/ # 350份人类标注文件（真实标签）
│ ├── dask__12221_human.json
│ └── ...
├── contexts/
│ ├── config_A/ # 350份预构建的config_A上下文
│ ├── config_B/ # 350份预构建的config_B上下文
│ └── config_C/ # 350份预构建的config_C上下文
└── evals/
└── eval_100.json # 论文中使用的100条拉取请求分层样本

## 文件格式

**`prs.jsonl`**：每条记录对应一条拉取请求，格式如下：
json
{
"task_id": "dask__12221",
"repo": "dask/dask",
"language": "Python",
"difficulty": "Type1_Direct",
"rvs_score": 0.52,
"diff_patch": "diff --git ...",
"base_commit": "0a075534...",
"head_commit": "59dab320...",
"num_substantive_comments": 3
}

**`annotations/dask__12221_human.json`**：真实标签文件，格式如下：
json
{
"task_id": "dask__12221",
"comments": [
{
"comment_id": "c_1",
"body": "超出范围：此内容应归属os.process_cpu_count...",
"file": "dask/system.py",
"line": 82,
"is_in_diff": true,
"is_initiating_comment": true
}
]
}

**`contexts/config_A/dask__12221.json`**：上下文文件，格式如下：
json
{
"task_id": "dask__12221",
"config_name": "config_A",
"pipeline_version": "v0.4.1",
"total_tokens": 847,
"was_truncated": false,
"rendered": "## Layer 0 - Task + Focus
..."
}

**`evals/eval_100.json`**：论文评估划分的数据集，格式如下：
json
{
"description": "论文基线实验使用的100条拉取请求分层样本。",
"n": 100,
"stratification": {
"Type1_Direct": 40,
"Type2_Contextual": 40,
"Type3_Latent": 20
},
"task_ids": ["dask__12221", "prowler__9865", "..."]
}

## 真实标签说明

真实标签均为人类工程师在实际审查流程中，针对真实合并拉取请求所撰写的审查评论，通过GitHub审查API事后收集得到。**在数据集构建过程中，未生成、合成或修改任何评论。**

真实标签的纳入标准（源自`RUBRIC.md`）：
- 人类撰写
- 为发起评论，而非回复
- 字数≥10
- 提及特定代码行为
- 非纯赞美评论

## 基准榜单提交

本数据集遵循SWE-Bench与SWE-Bench Pro确立的荣誉体系标准，公开所有350条拉取请求。评估工具与提交指南即将发布。若有意提交结果，请在此数据集页面开启讨论。

## 引用格式
bibtex
@article{kumar2026sweprbench,
title={SWE-PRBench: Benchmarking AI Code Review Quality
Against Real Pull Request Feedback},
author={Kumar, Deepak},
journal={arXiv preprint},
year={2026}
}

## 许可证

数据集许可证：CC BY 4.0