Benchmarking LLM-based Information Extraction Tools for Medical Documents

Motivation: Medical documents are a crucial resource for medical research around the world. While troves of valuable health data exist, they are largely computationally inaccessible as hard copies of unstructured text, often with degraded quality due to the persistent prevalence of fax machines in medical settings. Digitization of these resources through manual data extraction is time-consuming and resource intensive. However, large language models (LLMs) have recently shown great promise for automated digitization and information extraction (IE), greatly improving upon previous tools in terms of speed and accuracy.Results: We reviewed LLM-based IE tools from the literature and assessed them with respect to their suitability for use in biomedical research. We found only one of these tools (NuExtract2) to satisfy our selection criteria and compared it to LLM foundation models prompted to perform extractions. We created 200 mock medical documents with paired reference data using a bespoke procedural generation approach and evaluated the tools' performance across different prompting strategies, input modalities, and document qualities. We found model performances to be highly variable and very sensitive to input modalities and quality. Among the tested open-weights models, the proprietary GPT 4.1-mini performed the best on image inputs, with an average F1 score of 56.5%. The best performing local model was Gemma3 with an average F1 score of 55.6% on OCR text inputs. The NuExtract2 model was the only one to be able run on regular laptop computers, but did not perform well. Surprisingly, we found the choice of one-shot over zero-shot prompts to only have a small effect size on extraction performances in most cases. Availability: Source code and data are available on Github at https://github.com/courtotlab/extraction-benchmark