Annotated Benchmark for Evaluating Deduplication Tools in Evidence Synthesis

This dataset is decribed in details in publication Sadowska et. al. in press.

This annotated benchmark dataset is developed for the validation and comparative evaluation of deduplication tools used in evidence synthesis. The benchmark comprises three review-specific subsets derived from real-world systematic review searches covering 7–10 bibliographic databases and ranging from 3,645 to 31,740 records. Dataset development involved independent double manual deduplication, structured conflict resolution, automated algorithm-driven deduplication, and final adjudication of discrepancies between human and tool-based classifications. To ensure reproducibility and enable automated performance assessment, the dataset includes transparent annotations and predefined record-retention rules that unambiguously identify which records are retained and which are treated as duplicates. The benchmark is intended to support methodological research and standardized validation of deduplication tools in evidence synthesis.

The dataset contains files corresponding to three source systematic reviews, with file names based on the surname of the first author of each review. For each review, three RIS files are provided: a _main file containing all records retrieved from the databases following search replication, a _duplicates file containing records identified as duplicates, and a _non-duplicates file containing records retained after deduplication. The deduplicated files are annotated using the Custom1 field, which provides unique record identifiers and enables unambiguous determination of which records were retained and which were classified as duplicates. Detailed information on the annotation methodology and record-retention rules is available in the accompanying publication.

The dataset includes RIS files retrieved from the following databases: MEDLINE; Embase; PubMed; The Cochrane Central Register of Controlled Trials (CENTRAL); PsycINFO; Scientific Electronic Library Online (SciELO); Latin American and Caribbean Health Sciences Literature (LILACS); Web of Science; Scopus; ScienceDirect; Cochrane Database of Systematic Reviews; Science Citation Index – Web of Science; Conference Proceedings Citation Index – Science (Web of Science); Rehab; SpeechBITE; CINAHL; AgeLine.

By requesting access to, downloading, or using this dataset, the user confirms that they hold a valid and active license or authorized institutional access to all proprietary databases from which the records were originally retrieved.