thoughtworks/document-processing-benchmark
收藏资源简介:
--- license: other license_name: mixed-research-and-permissive license_link: LICENSE language: - en - id task_categories: - image-to-text - visual-question-answering - image-classification tags: - documents - receipts - invoices - forms - contracts - bank-statements - ocr - vlm - benchmark pretty_name: Document Processing Benchmark size_categories: - 1K<n<10K configs: - config_name: test data_files: - split: test path: data/test/docs.parquet - config_name: full data_files: - split: train path: data/full/docs.parquet - config_name: v2 data_files: - split: train path: data/v2/docs.parquet - config_name: v3 data_files: - split: train path: data/v3/docs.parquet --- # Document Processing Benchmark 8 public document datasets (receipts, invoices, forms, bank statements, multi-page docs, contracts) normalized into one parquet schema. Each row has the document, ground-truth annotations, and **per-row token/latency/cost numbers from real API calls** to one or more reference models. You can read off a target's cost/latency/quality without re-running it. ```python from datasets import load_dataset ds = load_dataset("thoughtworks/document-processing-benchmark", "v3", split="train") row = ds[0] row["baseline_gpt5mini_multi_image_prompt_tokens"] # real billed ISL from OpenAI row["baseline_gpt5mini_multi_image_completion_tokens"] row["baseline_gpt5mini_multi_image_e2el_ms"] ``` ## Pick a config by what you're measuring | config | n docs | size | what it covers | what's baselined | |---|--:|--:|---|---| | `test` | 4,522 | 2.7 GB | Single-page receipts/invoices/forms, full-schema (wide) extraction. **2:1 ISL:OSL** | gpt-5-mini, gpt-5.4-nano, gemma-4-26B (wide prompt) | | `full` | 10,074 | 6.4 GB | Same content as `test`, more docs per source | gpt-5-mini, gpt-5.4-nano, gemma-4-26B (wide prompt) | | `v2` | 900 | 8.1 GB | Multi-page VQA + bank statements, narrow (3-5 field) extraction. **17.6:1 ISL:OSL** | gpt-5-mini (narrow) + claude-sonnet-4-5 judge | | `v3` | 186 | 44 MB | Real multi-page contracts (CUAD). Raw PDFs per row. **3 preprocessing baselines side-by-side** | gpt-5-mini × {multi-image, native-pdf, text-only} preproc | **ISL = input tokens, OSL = output tokens.** Different configs target different production workload shapes — pick by the column you'd plot for your use case: - Single-page docs, full structured extraction → `test` / `full` - Multi-page docs or narrow header-extraction → `v2` - Multi-page docs where the preprocessing path matters → `v3` (cheapest path costs 3.4× less than the most expensive on identical docs) ## What's in every row 22 base columns shared across all configs (`doc_id`, `source_dataset`, `doc_type`, `image_b64`, image-token formulas for 5 vendors, `ground_truth_json`, license metadata, etc.). `v3` additionally has `pdf_b64` (raw PDF base64) and `num_pages`. Per-row baseline columns follow this convention: ``` baseline_<model_slug>_<variant_suffix>_<metric> ``` Where: - `<model_slug>` = lowercase no-dash model name (`gpt5mini`, `gpt54nano`, `gemma426ba4bit`) - `<variant_suffix>` = empty for v1 wide; `narrow` for v2; preproc name for v3 (`multi_image`, `native_pdf`, `text_only`) - `<metric>` = one of `prompt_tokens, completion_tokens, reasoning_tokens, visible_output_tokens, cached_tokens, e2el_ms, finish_reason, parse_failed, prediction_text` For v2 there are also `judge_score_gpt5mini_narrow_{full,narrow}` (float) and `judge_per_field_gpt5mini_narrow` (JSON) from claude-sonnet-4-5 as judge. See [`docs/doc-bench-SCHEMA.md`](https://github.com/tails-mpt/tw-taas/blob/feat/doc-bench-v2/docs/doc-bench-SCHEMA.md) in the source repo for the full column list. ## Sources by config | source | upstream | configs | doc_type | license | |---|---|---|---|---| | cord_v2 | [naver-clova-ix/cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) | test, full | receipt | CC-BY-4.0 | | funsd | [nielsr/funsd-layoutlmv3](https://huggingface.co/datasets/nielsr/funsd-layoutlmv3) | test, full | form | CDLA-Sharing-1.0 | | sroie | [mychen76/ds_receipts_v2_train](https://huggingface.co/datasets/mychen76/ds_receipts_v2_train) | test, full | receipt | ICDAR-2019 research-only | | wildreceipt | [kaydee/wildreceipt](https://huggingface.co/datasets/kaydee/wildreceipt) | test, full | receipt | Apache-2.0 | | fatura | [mathieu1256/FATURA2-invoices](https://huggingface.co/datasets/mathieu1256/FATURA2-invoices) | test, full | invoice | CC-BY-NC-4.0 | | rvl_cdip_invoice | [chainyo/rvl-cdip-invoice](https://huggingface.co/datasets/chainyo/rvl-cdip-invoice) | test, full | invoice | Research-only (CDIP) | | mp_docvqa | [lmms-lab/MP-DocVQA](https://huggingface.co/datasets/lmms-lab/MP-DocVQA) | v2 | vqa_multipage | MIT | | bank_statements_agami | [AgamiAI/Indian-Bank-Statements](https://huggingface.co/datasets/AgamiAI/Indian-Bank-Statements) | v2 | bank_statement | Apache-2.0 | | cuad | [theatticusproject/cuad](https://huggingface.co/datasets/theatticusproject/cuad) | v3 | contract | CC-BY-4.0 | ⚠️ **License notice.** Mix of permissive and research-only/non-commercial. For commercial use, filter to `source_dataset in {cord_v2, funsd, wildreceipt, mp_docvqa, bank_statements_agami, cuad}`. Per-row `license` column for programmatic filtering. ## Per-source signature (gpt-5-mini, real billed) Aggregated across all variants. Use to project cost onto your traffic mix: `cost = Σ (per-source $/1K × source share × daily volume / 1000)`. | variant | source | prompt | preproc | n | ISL p50 | ISL p95 | OSL p50 | ratio | $/doc | |---|---|---|---|--:|--:|--:|--:|--:|--:| | test (v1) | cord_v2 | wide | image | 100 | 1,383 | 2,660 | 277 | 4.6:1 | $0.00099 | | test (v1) | sroie | wide | image | 1,000 | 324 | 2,902 | 379 | 1.7:1 | $0.00110 | | test (v1) | wildreceipt | wide | image | 472 | 414 | 3,417 | 383 | 1.9:1 | $0.00112 | | test (v1) | fatura | wide | image | 1,400 | 679 | 679 | 356 | 1.9:1 | $0.00090 | | test (v1) | rvl_cdip_invoice | wide | image | 1,500 | 985 | 1,023 | 368 | 2.3:1 | $0.00111 | | test (v1) | funsd | wide | image | 50 | 954 | 1,031 | 631 | 1.4:1 | $0.00165 | | v2 | bank_statements_agami | narrow | image | 400 | 2,483 | 3,282 | 124 | 19.7:1 | $0.00101 | | v2 | mp_docvqa | narrow | image | 500 | 1,294 | 3,548 | 82 | 15.2:1 | $0.00064 | | v3 | cuad | narrow | multi-image | 186 | **24,442** | 97,642 | 193 | 159.7:1 | $0.00868 | | v3 | cuad | narrow | native-pdf | 186 | 6,995 | 33,634 | 229 | 43.7:1 | $0.00307 | | v3 | cuad | narrow | text-only | 186 | 6,075 | 24,171 | 198 | 39.4:1 | $0.00252 | The v3 multi-image row at ISL p50 24,442 matches a typical production accounting-feature shape (18-22K reported by one production deployment). Other rows let you slice down to smaller or simpler shapes. ## Evaluating your own model Minimal example: ```python from datasets import load_dataset from openai import OpenAI ds = load_dataset("thoughtworks/document-processing-benchmark", "v3", split="train") client = OpenAI(base_url="https://your-endpoint/v1", api_key="...") for row in ds: # multi-image preproc — render PDF pages and send N image parts pdf_b64 = row["pdf_b64"] # ... render PDF → list of base64 PNGs ... resp = client.chat.completions.create( model="your-model", messages=[{"role": "user", "content": [ *[{"type":"image_url","image_url":{"url":f"data:image/png;base64,{p}","detail":"high"}} for p in pages], {"type":"text","text":"Extract parties, contract_type, effective_date, termination_date. JSON only."}, ]}], ) # Compare row-by-row: your_isl = resp.usage.prompt_tokens ref_isl = row["baseline_gpt5mini_multi_image_prompt_tokens"] ``` A ready-made harness with all 4 preproc paths + augment + report scripts lives at [github.com/tails-mpt/tw-taas](https://github.com/tails-mpt/tw-taas) on the `feat/doc-bench-v2` branch. See [`docs/doc-bench-EXTENDING.md`](https://github.com/tails-mpt/tw-taas/blob/feat/doc-bench-v2/docs/doc-bench-EXTENDING.md) for the full how-to. ## Caveats - **Image-token billing for gpt-5-mini ≠ the published gpt-4o formula.** The in-row `image_tokens_openai_high` column is a useful *shape* signal but underestimates real billing by a factor of ~33×. Use the captured `baseline_<model>_*_prompt_tokens` from a real API call for absolute cost. - **bank_statements_agami is synthetic** (4 layout templates × 100 docs). ISL distribution is bimodal — not a continuous real-world spread. - **AgamiAI bank statements are first-page-only renders.** Real bank statements span multiple pages; treat this source as "single-page bank statement layout" only. - **FATURA is template-generated.** Useful for controlled-environment benchmarking; less so for messy real-world variation. - **CUAD page cap = 50.** The 13 longest contracts (>50 pages) were dropped to bound cost-per-call. ## Citation Cite the upstream sources per their published references: - CORD-v2: Park et al., 2019 (Naver Clova) - FUNSD: Jaume et al., 2019 - SROIE: Huang et al., 2019 (ICDAR) - WildReceipt: Sun et al., 2021 - FATURA: Limam et al., 2023 - RVL-CDIP: Harley et al., 2015 - MP-DocVQA: Tito et al., 2022 - CUAD: Hendrycks et al., 2021 This benchmark assembly is released under the union of upstream licenses listed in the Sources section. Filter by the per-row `license` column for your use case (commercial vs research).
Document Processing Benchmark is a benchmark dataset for document processing that normalizes 8 public document datasets (receipts, invoices, forms, bank statements, multi-page docs, contracts) into one parquet schema. Each row has the document, ground-truth annotations, and per-row token/latency/cost numbers from real API calls to one or more reference models. You can read off a targets cost/latency/quality without re-running it. The dataset offers multiple configs (e.g., test, full, v2, v3) targeting different production workload shapes, such as single-page full structured extraction, multi-page or narrow header extraction, and preprocessing path comparisons. Every row includes 22 base columns (e.g., doc_id, source_dataset, doc_type, image_b64, ground_truth_json) and baseline columns following a naming convention for model variants. Sources are diverse with a mix of permissive and research-only licenses, requiring filtering by the per-row license column for appropriate use cases.




