mercor/apex-agents

--- license: cc-by-4.0 language: - en tags: - agents - benchmarking - finance - legal - management-consulting - tool-use - long-horizon pretty_name: apex-agents size_categories: - n<1K --- # APEX–Agents APEX–Agents is a benchmark from [Mercor](https://www.mercor.com/apex/) for evaluating whether AI agents can execute long-horizon, cross-application professional services tasks. Tasks were created by **investment banking analysts**, **management consultants**, and **corporate lawyers**, and require agents to navigate realistic work environments with files and tools (e.g., docs, spreadsheets, PDFs, email, chat, calendar). - **Tasks:** 480 total (160 per job category) - **Worlds:** 33 total (10 banking, 11 consulting, 12 law) - **Rubric criteria:** binary, criterion-level grading; mean ~4 criteria per task - **Gold outputs:** provided for every task - **World assets:** included (files + metadata) - **License:** CC-BY 4.0 - **Intended use:** APEX-Agents is intended exclusively for model evaluation. Any use of this dataset for training, fine-tuning, or parameter fitting is expressly forbidden. ## Dataset overview |Job | # Worlds | Avg files / world | # Tasks | Avg criteria / task | Avg est. hours | Tasks w/ file outputs | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | Investment banking | 10 | 172 | 160 | 2.93 | 1.36 | 27 (16.9%) | | Law | 12 | 161 | 160 | 4.57 | 2.40 | 20 (12.5%) | | Management consulting | 11 | 165 | 160 | 4.68 | 1.69 | 11 (6.9%) | | **Benchmark total** | **33** | **166** | **480** | **4.06** | **1.82** | **58 (12.1%)** | Each case corresponds to a **task** inside a **world**. A “world” is a realistic project scenario created by experts. Worlds contain files and tools required to complete tasks. Web search is disabled to keep evaluations reproducible. Worlds contain applications such as: Calendar, Chat, Code Execution, Documents, File system, Mail, PDFs, Spreadsheets, Presentations. Some worlds include additional finance data applications. A task includes: - **Prompt**: single-turn instruction given to the agent - **Rubric**: list of criteria (binary gradable statements) + grading target info - **Gold output(s)**: expert-created reference output (in the requested output format) - **Metadata**: job/workflow tags, expected output type, estimated completion time, etc. - **World context**: pointers/IDs for the world plus associated files/artifacts ## Evaluation APEX–Agents uses **rubric-based grading**: - Each rubric contains multiple **criteria** (binary: Met / Not met). - There are between 1 and 10 criteria, with a mean of 4.06. - A judge model grades each criterion independently, using the prompt, the agent output, and relevant artifacts/changes. ## Leaderboard baselines Our paper reports results for eight agents against multiple metrics (Pass@1, Pass@8, mean score). The leaderboard uses **Pass@1**: the probability that a uniformly sampled task passes all criteria in a single run. Where available, models have thinking / reasoning effort set to **high**. | Model | Pass@1 (95% CI) | Pass@8 (95% CI) | Pass^8 | Mean score | IB analyst Pass@1 | Consultant Pass@1 | Lawyer Pass@1 | |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | Claude Opus 4.5 | 18.4% \[15.5–21.3] | 34.0% \[29.8–38.3] | 8.8% | 34.8% | 21.6% | 13.2% | 20.2% | | Gemini 3 Flash | 24.0% \[20.7–27.3] | 36.7% \[32.3–41.0] | 13.4% | 39.5% | 26.7% | 19.3% | 25.9% | | Gemini 3 Pro | 18.4% \[15.7–21.1] | 37.3% \[32.9–41.7] | 6.5% | 34.1% | 18.8% | 12.4% | 23.9% | | GPT-5 | 18.3% \[15.4–21.3] | 31.0% \[26.9–35.4] | 7.7% | 32.9% | 27.3% | 12.3% | 15.3% | | GPT-5.2 | 23.0% \[19.8–26.2] | 40.0% \[35.6–44.4] | 11.0% | 38.7% | 27.3% | 22.7% | 18.9% | | GPT-OSS-120B | 4.7% \[3.3–6.1] | 11.5% \[8.8–14.4] | 1.2% | 14.5% | 2.7% | 3.5% | 7.8% | | Grok 4 | 15.2% \[12.8–17.7] | 32.9% \[28.7–37.3] | 4.7% | 30.3% | 17.0% | 12.0% | 16.5% | | Kimi K2 Thinking | 4.0% \[2.9–5.2] | 14.4% \[11.5–17.5] | 0.3% | 11.5% | 1.2% | 2.9% | 8.0% | ## Archipelago Our service for executing and evalling agents is available open-source on Github. ✨[View the code](https://github.com/Mercor-Intelligence/archipelago) ## How to load the dataset ```python from datasets import load_dataset ds = load_dataset("mercor/apex-agents") # replace if your org/name differs print(ds) print(ds["train"][0].keys()) ``` ## Citation ```bibtex @misc{vidgen2026apexagents, title = {APEX--Agents}, author = {Vidgen, Bertie and Mann, Austin and Fennelly, Abby and Wright Stanly, John and Rothman, Lucas and Burstein, Marco and Benchek, Julien and Ostrofsky, David and Ravichandran, Anirudh and Sur, Debnil and Venugopal, Neel and Hsia, Alannah and Robinson, Isaac and Huang, Calix and Varones, Olivia and Khan, Daniyal and Haines, Michael and Richards, Zach and Mahapatra, Chirag and Foody, Brendan and Nitski, Osvald}, year = {2026}, howpublished = {arXiv}, url = {https://arxiv.org/pdf/2601.14242} } ``` ## Contact us [[e-mail]](mailto:[e-mail]) ## Legal disclaimer on the content of worlds This material is provided for research, educational, and informational purposes only. It consists of hypothetical, simulated financial and legal and regulatory analyses and illustrative scenarios (including, without limitation, simulated leverage buyout structures, capital structures, financing terms, valuation ranges, projected returns, and potential mergers, acquisitions, divestitures, or other strategic transactions, legal memoranda, hypothetical legal advice to a company, hypothetical correspondences to regulatory agencies, etc.). No representation is made that any scenario described herein is likely to occur, is being contemplated by any person, or reflects an actual proposed or pending transaction or any legal, regulatory, or compliance risk. This material does not constitute (and should not be construed as) financial, investment, legal, tax, accounting, or other professional advice, and is not intended to form the basis of any investment decision or any contract. The analyses and outputs in this material are based on assumptions, estimates, modeling methodologies, and hypothetical legal scenarios, that may prove incorrect. The financial and legal information is derived from publicly available information and third‑party sources that have not been independently verified. Any projections, forward‑looking statements, scenario outputs, similar financial information, and any legal documents, memoranda, and correspondence, are hypothetical and thus inherently uncertain and are provided solely to illustrate how results might change under different assumptions. No representation or warranty (express or implied) is made regarding this material, and it is provided on an “as‑is” and “as‑available” basis. To the maximum extent permitted by applicable law, Mercor disclaims any liability for any direct or indirect losses or damages arising from or related to the use of (or reliance on) this material, including without limitation any loss of profits, loss of business, loss of goodwill, or consequential, incidental, special, punitive, or exemplary damages, even if advised of the possibility of such damages. Nothing in this disclaimer limits or excludes liability that cannot be limited or excluded under applicable law.APEX–Agents is a benchmark from Mercor for evaluating whether AI agents can execute long-horizon, cross-application professional services tasks. Tasks were created by investment banking analysts, management consultants, and corporate lawyers, and require agents to navigate realistic work environments with files and tools (e.g., docs, spreadsheets, PDFs, email, chat, calendar). The dataset includes 480 total tasks (160 per job category), 33 total worlds (10 banking, 11 consulting, 12 law), rubric criteria with a mean of ~4 criteria per task, gold outputs for every task, and world assets (files + metadata). The dataset is licensed under CC-BY 4.0 and is intended exclusively for model evaluation. Any use of this dataset for training, fine-tuning, or parameter fitting is expressly forbidden.

搜集汇总

背景与挑战

背景概述

APEX–Agents是一个基准数据集，用于评估AI代理在真实工作环境中执行长期、跨应用的专业服务任务，涵盖投资银行、管理咨询和公司法律三个职业类别。数据集包含480个任务、33个世界（每个职业约160个任务），并提供每个任务的评分标准和黄金输出，采用CC-BY 4.0许可，仅限模型评估使用，禁止用于训练或微调。