遇见数据集

thoughtworks/document-processing-benchmark

收藏
Hugging Face2026-05-24 更新2026-06-14 收录
官方服务:

资源简介:

--- license: other license_name: mixed-research-and-permissive license_link: LICENSE language: - en - id task_categories: - image-to-text - visual-question-answering - image-classification tags: - documents - receipts - invoices - forms - contracts - bank-statements - ocr - vlm - benchmark pretty_name: Document Processing Benchmark size_categories: - 1K<n<10K configs: - config_name: test data_files: - split: test path: data/test/docs.parquet - config_name: full data_files: - split: train path: data/full/docs.parquet - config_name: v2 data_files: - split: train path: data/v2/docs.parquet - config_name: v3 data_files: - split: train path: data/v3/docs.parquet --- # Document Processing Benchmark 8 public document datasets (receipts, invoices, forms, bank statements, multi-page docs, contracts) normalized into one parquet schema. Each row has the document, ground-truth annotations, and **per-row token/latency/cost numbers from real API calls** to one or more reference models. You can read off a target's cost/latency/quality without re-running it. ```python from datasets import load_dataset ds = load_dataset("thoughtworks/document-processing-benchmark", "v3", split="train") row = ds[0] row["baseline_gpt5mini_multi_image_prompt_tokens"] # real billed ISL from OpenAI row["baseline_gpt5mini_multi_image_completion_tokens"] row["baseline_gpt5mini_multi_image_e2el_ms"] ``` ## Pick a config by what you're measuring | config | n docs | size | what it covers | what's baselined | |---|--:|--:|---|---| | `test` | 4,522 | 2.7 GB | Single-page receipts/invoices/forms, full-schema (wide) extraction. **2:1 ISL:OSL** | gpt-5-mini, gpt-5.4-nano, gemma-4-26B (wide prompt) | | `full` | 10,074 | 6.4 GB | Same content as `test`, more docs per source | gpt-5-mini, gpt-5.4-nano, gemma-4-26B (wide prompt) | | `v2` | 900 | 8.1 GB | Multi-page VQA + bank statements, narrow (3-5 field) extraction. **17.6:1 ISL:OSL** | gpt-5-mini (narrow) + claude-sonnet-4-5 judge | | `v3` | 186 | 44 MB | Real multi-page contracts (CUAD). Raw PDFs per row. **3 preprocessing baselines side-by-side** | gpt-5-mini × {multi-image, native-pdf, text-only} preproc | **ISL = input tokens, OSL = output tokens.** Different configs target different production workload shapes — pick by the column you'd plot for your use case: - Single-page docs, full structured extraction → `test` / `full` - Multi-page docs or narrow header-extraction → `v2` - Multi-page docs where the preprocessing path matters → `v3` (cheapest path costs 3.4× less than the most expensive on identical docs) ## What's in every row 22 base columns shared across all configs (`doc_id`, `source_dataset`, `doc_type`, `image_b64`, image-token formulas for 5 vendors, `ground_truth_json`, license metadata, etc.). `v3` additionally has `pdf_b64` (raw PDF base64) and `num_pages`. Per-row baseline columns follow this convention: ``` baseline_<model_slug>_<variant_suffix>_<metric> ``` Where: - `<model_slug>` = lowercase no-dash model name (`gpt5mini`, `gpt54nano`, `gemma426ba4bit`) - `<variant_suffix>` = empty for v1 wide; `narrow` for v2; preproc name for v3 (`multi_image`, `native_pdf`, `text_only`) - `<metric>` = one of `prompt_tokens, completion_tokens, reasoning_tokens, visible_output_tokens, cached_tokens, e2el_ms, finish_reason, parse_failed, prediction_text` For v2 there are also `judge_score_gpt5mini_narrow_{full,narrow}` (float) and `judge_per_field_gpt5mini_narrow` (JSON) from claude-sonnet-4-5 as judge. See [`docs/doc-bench-SCHEMA.md`](https://github.com/tails-mpt/tw-taas/blob/feat/doc-bench-v2/docs/doc-bench-SCHEMA.md) in the source repo for the full column list. ## Sources by config | source | upstream | configs | doc_type | license | |---|---|---|---|---| | cord_v2 | [naver-clova-ix/cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) | test, full | receipt | CC-BY-4.0 | | funsd | [nielsr/funsd-layoutlmv3](https://huggingface.co/datasets/nielsr/funsd-layoutlmv3) | test, full | form | CDLA-Sharing-1.0 | | sroie | [mychen76/ds_receipts_v2_train](https://huggingface.co/datasets/mychen76/ds_receipts_v2_train) | test, full | receipt | ICDAR-2019 research-only | | wildreceipt | [kaydee/wildreceipt](https://huggingface.co/datasets/kaydee/wildreceipt) | test, full | receipt | Apache-2.0 | | fatura | [mathieu1256/FATURA2-invoices](https://huggingface.co/datasets/mathieu1256/FATURA2-invoices) | test, full | invoice | CC-BY-NC-4.0 | | rvl_cdip_invoice | [chainyo/rvl-cdip-invoice](https://huggingface.co/datasets/chainyo/rvl-cdip-invoice) | test, full | invoice | Research-only (CDIP) | | mp_docvqa | [lmms-lab/MP-DocVQA](https://huggingface.co/datasets/lmms-lab/MP-DocVQA) | v2 | vqa_multipage | MIT | | bank_statements_agami | [AgamiAI/Indian-Bank-Statements](https://huggingface.co/datasets/AgamiAI/Indian-Bank-Statements) | v2 | bank_statement | Apache-2.0 | | cuad | [theatticusproject/cuad](https://huggingface.co/datasets/theatticusproject/cuad) | v3 | contract | CC-BY-4.0 | ⚠️ **License notice.** Mix of permissive and research-only/non-commercial. For commercial use, filter to `source_dataset in {cord_v2, funsd, wildreceipt, mp_docvqa, bank_statements_agami, cuad}`. Per-row `license` column for programmatic filtering. ## Per-source signature (gpt-5-mini, real billed) Aggregated across all variants. Use to project cost onto your traffic mix: `cost = Σ (per-source $/1K × source share × daily volume / 1000)`. | variant | source | prompt | preproc | n | ISL p50 | ISL p95 | OSL p50 | ratio | $/doc | |---|---|---|---|--:|--:|--:|--:|--:|--:| | test (v1) | cord_v2 | wide | image | 100 | 1,383 | 2,660 | 277 | 4.6:1 | $0.00099 | | test (v1) | sroie | wide | image | 1,000 | 324 | 2,902 | 379 | 1.7:1 | $0.00110 | | test (v1) | wildreceipt | wide | image | 472 | 414 | 3,417 | 383 | 1.9:1 | $0.00112 | | test (v1) | fatura | wide | image | 1,400 | 679 | 679 | 356 | 1.9:1 | $0.00090 | | test (v1) | rvl_cdip_invoice | wide | image | 1,500 | 985 | 1,023 | 368 | 2.3:1 | $0.00111 | | test (v1) | funsd | wide | image | 50 | 954 | 1,031 | 631 | 1.4:1 | $0.00165 | | v2 | bank_statements_agami | narrow | image | 400 | 2,483 | 3,282 | 124 | 19.7:1 | $0.00101 | | v2 | mp_docvqa | narrow | image | 500 | 1,294 | 3,548 | 82 | 15.2:1 | $0.00064 | | v3 | cuad | narrow | multi-image | 186 | **24,442** | 97,642 | 193 | 159.7:1 | $0.00868 | | v3 | cuad | narrow | native-pdf | 186 | 6,995 | 33,634 | 229 | 43.7:1 | $0.00307 | | v3 | cuad | narrow | text-only | 186 | 6,075 | 24,171 | 198 | 39.4:1 | $0.00252 | The v3 multi-image row at ISL p50 24,442 matches a typical production accounting-feature shape (18-22K reported by one production deployment). Other rows let you slice down to smaller or simpler shapes. ## Evaluating your own model Minimal example: ```python from datasets import load_dataset from openai import OpenAI ds = load_dataset("thoughtworks/document-processing-benchmark", "v3", split="train") client = OpenAI(base_url="https://your-endpoint/v1", api_key="...") for row in ds: # multi-image preproc — render PDF pages and send N image parts pdf_b64 = row["pdf_b64"] # ... render PDF → list of base64 PNGs ... resp = client.chat.completions.create( model="your-model", messages=[{"role": "user", "content": [ *[{"type":"image_url","image_url":{"url":f"data:image/png;base64,{p}","detail":"high"}} for p in pages], {"type":"text","text":"Extract parties, contract_type, effective_date, termination_date. JSON only."}, ]}], ) # Compare row-by-row: your_isl = resp.usage.prompt_tokens ref_isl = row["baseline_gpt5mini_multi_image_prompt_tokens"] ``` A ready-made harness with all 4 preproc paths + augment + report scripts lives at [github.com/tails-mpt/tw-taas](https://github.com/tails-mpt/tw-taas) on the `feat/doc-bench-v2` branch. See [`docs/doc-bench-EXTENDING.md`](https://github.com/tails-mpt/tw-taas/blob/feat/doc-bench-v2/docs/doc-bench-EXTENDING.md) for the full how-to. ## Caveats - **Image-token billing for gpt-5-mini ≠ the published gpt-4o formula.** The in-row `image_tokens_openai_high` column is a useful *shape* signal but underestimates real billing by a factor of ~33×. Use the captured `baseline_<model>_*_prompt_tokens` from a real API call for absolute cost. - **bank_statements_agami is synthetic** (4 layout templates × 100 docs). ISL distribution is bimodal — not a continuous real-world spread. - **AgamiAI bank statements are first-page-only renders.** Real bank statements span multiple pages; treat this source as "single-page bank statement layout" only. - **FATURA is template-generated.** Useful for controlled-environment benchmarking; less so for messy real-world variation. - **CUAD page cap = 50.** The 13 longest contracts (>50 pages) were dropped to bound cost-per-call. ## Citation Cite the upstream sources per their published references: - CORD-v2: Park et al., 2019 (Naver Clova) - FUNSD: Jaume et al., 2019 - SROIE: Huang et al., 2019 (ICDAR) - WildReceipt: Sun et al., 2021 - FATURA: Limam et al., 2023 - RVL-CDIP: Harley et al., 2015 - MP-DocVQA: Tito et al., 2022 - CUAD: Hendrycks et al., 2021 This benchmark assembly is released under the union of upstream licenses listed in the Sources section. Filter by the per-row `license` column for your use case (commercial vs research).

Document Processing Benchmark is a benchmark dataset for document processing that normalizes 8 public document datasets (receipts, invoices, forms, bank statements, multi-page docs, contracts) into one parquet schema. Each row has the document, ground-truth annotations, and per-row token/latency/cost numbers from real API calls to one or more reference models. You can read off a targets cost/latency/quality without re-running it. The dataset offers multiple configs (e.g., test, full, v2, v3) targeting different production workload shapes, such as single-page full structured extraction, multi-page or narrow header extraction, and preprocessing path comparisons. Every row includes 22 base columns (e.g., doc_id, source_dataset, doc_type, image_b64, ground_truth_json) and baseline columns following a naming convention for model variants. Sources are diverse with a mix of permissive and research-only licenses, requiring filtering by the per-row license column for appropriate use cases.

提供机构:
thoughtworks
搜集汇总
数据集介绍
thoughtworks/document-processing-benchmark 数据集图片
构建方式
文档处理基准的构建依托于对现有公开文档数据集的系统化整合与归一化。研究者从收据、发票、表单、银行对账单、多页文档及合同等八类来源中采集原始数据,涵盖CORD-v2、FUNSD、SROIE、WildReceipt、FATURA、RVL-CDIP、MP-DocVQA与CUAD等经典语料。所有文档统一映射至单一Parquet模式,每行不仅包含文档图像与真实标注,还记录了针对一个或多个参考模型的实际API调用所产生token、延迟与成本数据。该设计使研究者无需重复运行模型即可读取目标模型的成本、延迟与质量指标,从而在统一框架下实现跨来源、跨任务形态的公平比较。
特点
该基准以多配置结构呈现差异化的工作负载形态,涵盖单页全模式抽取、多页视觉问答与窄字段抽取以及真实多页合同处理。每个配置均提供逐行基线指标,包括输入token、输出token、推理token、端到端延迟、解析失败标记及预测文本等,并遵循统一的列命名约定。基准还提供每来源的签名统计,如token分位数与每文档成本,便于将成本投影至实际流量组合。许可证元数据逐行标注,支持按商业或研究用途进行程序化过滤。值得注意的是,图像token计费公式与公开公式存在差异,须以真实API捕获的token数为准。
使用方法
使用者可通过Hugging Face数据集加载接口按配置名称获取数据,例如加载v3配置的训练集以处理多页合同。每行数据包含文档图像或原始PDF的base64编码、真实标注JSON以及基线模型的token与延迟指标。评估自有模型时,可渲染PDF页面为图像并发送至兼容OpenAI接口的端点,随后逐行比较所生成token与参考基线。配套的评估工具链涵盖四种预处理路径、数据增强与报告脚本,托管于开源仓库中。研究者可根据自身用例选择配置:单页全结构化抽取对应test或full,多页或窄字段抽取对应v2,预处理路径影响显著的多页文档对应v3。
背景与挑战
背景概述
文档智能处理作为连接物理文档与数字信息的关键环节,长期面临异构数据源难以统一评测的困境。2024年,Thoughtworks团队构建了document-processing-benchmark数据集,将CORD-v2、FUNSD、SROIE、WildReceipt、FATURA、RVL-CDIP、MP-DocVQA及CUAD等八个公开文档数据集归一化为统一parquet模式,涵盖收据、发票、表单、银行对账单、多页文档与合同等类型。该数据集的核心研究问题在于:如何在不重复调用商业API的前提下,系统性地量化视觉语言模型在真实文档处理工作负载中的质量、成本与延迟权衡。其独特之处在于每行均嵌入了来自gpt-5-mini等参考模型的真实计费token数、端到端延迟与成本数据,为文档理解领域提供了可复用的经济性基准,对推动OCR与VLM在产业场景中的落地评测具有显著影响力。
当前挑战
该数据集所应对的领域挑战源于文档处理任务的固有复杂性:文档版式多样、字段结构宽窄不一、多页合同与窄字段抽取的输入输出比悬殊,传统基准难以同时覆盖单页宽模式抽取与多页长文档理解。构建过程中亦面临多重挑战,包括混合许可证的合规性过滤、不同来源数据格式的归一化对齐、真实API计费token与公开图像token公式之间的显著偏差(约33倍)、合成银行对账单带来的分布不连续、FATURA模板生成缺乏真实杂乱性,以及CUAD长合同因成本上限而被截断至50页所引入的样本偏差。此外,多页PDF预处理路径的多样性(多图、原生PDF、纯文本)进一步增加了评测维度,使得成本与质量的权衡分析更为复杂。
常用场景
经典使用场景
在文档智能处理领域,面向票据、发票、表单、银行流水与合同等异构文档的统一评测长期受制于数据格式分散与标注口径不一。document-processing-benchmark将CORD-v2、FUNSD、SROIE、WildReceipt、FATURA、RVL-CDIP-Invoice、MP-DocVQA、Bank-Statements-Agami与CUAD等八个公开来源规范化为统一parquet模式,构成端到端基准。其最经典用法是选取test或full配置,对单页票据、发票与表单执行全模式结构化抽取,在真实调用所记录的输入输出token数、端到端时延与解析成功率层面,横向比较不同视觉语言模型的质量、成本与延迟,从而在无需重复调用参考模型的前提下完成目标模型的性能画像。
解决学术问题
该基准直面文档理解研究中长期存在的可复现性与可比性难题。以往工作多依赖单一数据集或自建评测,难以在统一条件下衡量模型在跨域文档上的信息抽取、视觉问答与分类能力,亦缺乏对推理开销的系统刻画。本数据集通过共享的22个基础列、逐行真值标注与真实计费统计,为跨文档类型迁移、宽窄模式抽取权衡、多页合同预处理路径比较等议题提供了可复用的实验底座,使得模型能力评估从孤立指标转向质量、成本与时延的多维权衡,推动了文档智能评测从碎片化走向标准化。
衍生相关工作
围绕该基准的组装与扩展,衍生出若干相关研究与工程工作。其配套github.com/tails-mpt/tw-taas仓库提供了覆盖全部预处理路径的评测框架与增强、报告脚本,降低了后续研究的复现门槛。基于CUAD子集的多页合同预处理对比,催生了关于原生PDF解析与多图渲染成本效益的系统性探讨。借鉴CORD-v2、FUNSD、SROIE等上游数据的既有工作,该基准进一步被用于视觉语言模型在杂乱真实文档上的鲁棒性检验,并为文档级基准的许可证合规与商业化过滤提供了可操作范式。
以上内容由遇见数据集搜集并总结生成
二维码
社区交流群
二维码
科研交流群
商业服务