Automating pharmacovigilance evidence generation: Using large language models to produce context-aware SQL

Objective: To enhance the accuracy of information retrieval from pharmacovigilance (PV) databases by employing Large Language Models (LLMs) to convert natural language queries (NLQs) into Structured Query Language (SQL) queries, leveraging a business context document. Materials and Methods: We utilized OpenAI’s GPT-4 model within a retrieval-augmented generation (RAG) framework, enriched with a business context document, to transform NLQs into executable SQL queries. Each NLQ was presented to the LLM randomly and independently to prevent memorization. The study was conducted in three phases, varying query complexity, and assessing the LLM's performance both with and without the business context document. Results: Our approach significantly improved NLQ-to-SQL accuracy, increasing from 8.3% with the database schema alone to 78.3% with the business context document. This enhancement was consistent across low, medium, and high complexity queries, indicating the critical role of contextual ..., Test set of NLQ's used in the paper Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL. Also included are the Python scripts for the LLM processing, the R code for statistical analysis of results, and a copy of the business context document and essential tables., , # Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL [https://doi.org/10.5061/dryad.2280gb63n](https://doi.org/10.5061/dryad.2280gb63n) ## Description of the data and file structure NLQ_Queries.xls contains the set of test NLQs along with the results of the LLM response in each phase of the experiment. Each NLQ also contains the complexity scores computed for each. The business context document is supplied as a PDF, together with the Python and R code used to generate our results. The essential tables used in Phase 2 and 3 of the experiment are included in the text file. ### Files and variables #### File: NLQ\_Queries.xlsx **Description:** Contains all NLQ queries with the results of the LLM output and the pass, fail status of each. **Column Definitions:** Below are the column names in order with a detailed description. 1. User NLQ: Plain text database query 2. Phase_1: Pass or Fail status indicator \"Pass, Partial, or Fa...### 研究目标 本研究旨在借助业务上下文文档，利用大语言模型（Large Language Model, LLM）将自然语言查询（natural language query, NLQ）转换为结构化查询语言（Structured Query Language, SQL），以提升药物警戒（pharmacovigilance, PV）数据库的信息检索准确率。 ### 材料与方法 本研究在检索增强生成（retrieval-augmented generation, RAG）框架内调用OpenAI的GPT-4模型，并辅以业务上下文文档，实现自然语言查询到可执行SQL查询的转换。为避免模型出现记忆现象，所有自然语言查询均以随机独立的方式输入大语言模型。本研究分为三个阶段，通过调整查询复杂度，分别测试有无业务上下文文档时大语言模型的性能表现。 ### 研究结果 本方法可显著提升自然语言查询到SQL的转换准确率：仅使用数据库架构时准确率为8.3%，加入业务上下文文档后提升至78.3%。该性能提升在低、中、高复杂度的查询中均保持一致，证明了上下文信息的关键作用……本研究使用的自然语言查询测试集来自论文《自动化药物警戒证据生成：利用大语言模型生成上下文感知SQL（Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL）》。此外，本数据集还包含用于大语言模型处理的Python脚本、用于结果统计分析的R代码，以及业务上下文文档与核心表格副本。 论文DOI链接：[https://doi.org/10.5061/dryad.2280gb63n](https://doi.org/10.5061/dryad.2280gb63n) --- ## 数据与文件结构说明 `NLQ_Queries.xls` 包含本次实验各阶段的自然语言查询测试集，以及大语言模型的响应结果，每条自然语言查询均附带计算得到的复杂度评分。 业务上下文文档以PDF格式提供，同时附带用于生成研究结果的Python与R代码。实验第二、三阶段所用的核心表格已保存至文本文件中。 ### 文件与变量 #### 文件：NLQ_Queries.xlsx **文件说明**：包含所有自然语言查询，以及大语言模型的输出结果与每条查询的通过/失败状态。 **列定义**： 按顺序列出各列名称与详细说明： 1. User NLQ：纯文本形式的数据库查询 2. Phase_1：通过/失败状态标识，可取值为"Pass、Partial 或 Fa..."