Progressively Refined Prompt Dataset for Legal Clause Extraction

Summary

This dataset has a collection of curated prompt templates that have been used to progressively enhance the clause extraction and document classification performance in legal contracts. These prompts were specifically designed for the ClauseMiner project, legal tech, and AI pipeline to automate the extraction process of relevant clauses and to label them with correct clause types with high accuracy. These collections of prompts have been explored thoroughly and evolved using different strategies, which range from general instructions to advanced chain-of-thought, role-based, iterative refinement, and few-shot prompting methodologies.

Motivation

The motivation behind these prompting techniques is to understand how different prompting strategies, as explained by Google's Whitepaper technique, influence the LLM response performance. It is extremely crucial in developing highly robust foundational legal AI models. This dataset provides users from the legal tech industry by giving them a traceable, methodological evolution of prompt designs. Each of these prompts has been optimized for structure, fidelity, and legal clarity. It is highly valuable, not only for clause extraction but is readily reusable for any legal-tech task that involves natural language processing and legal contract understanding. 

Content

prompt_structure.yaml - General structure for writing prompts.

prompt_general.yaml – General instructions for clause extraction.

prompt_contextual.yaml – Prompts leveraging contextual contract understanding.

prompt_constraint_based.yaml – Enforces strict legal content fidelity.

prompt_chain_of_thought.yaml – Introduces step-by-step clause mining logic.

prompt_iterative_refinement.yaml – Promotes multi-pass clause refinement.

prompt_few_shot.yaml – Uses multiple examples to guide learning.

prompt_one_shot.yaml – Uses a single example for generalization.

prompt_multi_task.yaml – Combines multiple legal tasks in one prompt.

prompt_role_based.yaml – Emulates expert legal analyst behavior.

prompt_mega.yaml – All-in-one, high-capacity expert prompt for top-tier extraction.

Each YAML file includes a structure template with proper placeholders for document inputs like {contract_text} and produces standardized outputs from various LLMs like LLaMA, OpenAI, Gemini, Sonar, etc., using clear clause naming and formatting conventions. 

Steps to Reproduce

Select Prompt Template: Choose the appropriate prompt template from the given dataset based on the complexity of your task. For eg. 

Use prompt_general.yaml or prompt_constraint_based.yaml for baseline tasks. 

Use prompt_chain_of_thought.yaml for creating a tree-based structure for deeper understanding tasks. 

Use prompt_iterative_refinement.yaml or prompt_few_shot.yaml for more nuanced document extraction. 

Prepare Contract Input: Insert the appropriate contract or agreement text inside a suitable place in your directory, and send the input to {contract_text}.

Use LLM Interface: Feed the readymade prompt along with the contract text into a large language model (LLM), such as OpenAI, Claude, LLaMA, or any other local LLMs via LangChain or other frameworks. 

Postprocess the Output: The output given by these LLMs is generally in the JSON format and mostly stored in the response key. Further, we can use regular expressions or script-based preprocessing to structure the output in the intended way into a string or CSV format for integrations into the downstream tasks. 

Evaluate and Iterate: Post received the outputs, and put a framework to calculate the accuracy. We can also measure the accuracy using a legal expert review. Then, switch between different prompt strategies or build upon the existing ones to observe the performance variation. 

General Use Cases

Clause extraction and document classification in AI contract review tools.

Advanced prompt engineering research in the legal NLP industry. 

Benchmarking for structure and few-shot learning based extraction methods.

Data preprocessing for retrieval-augmented generation (RAG) pipelines.

Explainability studies in the foundational legal tech AI models.