Smart-Contract-Vulnerability-Using-LLM

Smart Contract LLM Evaluation

Replication study of Salzano et al. (2026). This project replaces human expert review with locally hosted Large Language Models (LLMs) to evaluate how developers fix smart contract vulnerabilities in Solidity.

Requirements

Python 3.11 or higher

Ollama installed and running (https://ollama.com)

Paper dataset cloned from Zenodo

Install models:

ollama pull qwen2.5-coder:7b

ollama pull deepseek-r1:7

Clone dataset:

git clone https://zenodo.org/records/17105939 ~/smart-contract-eval/repo

Install Python dependencies:

pip install requests tqdm --break-system-packages

Folder structure

~/smart-contract-eval/

    repo/          dataset cloned from Zenodo

    scripts/       all scripts go here

    data/          intermediate JSONL files (auto-created)

    results/       final reports (auto-created)

Place all scripts inside ~/smart-contract-eval/scripts/.

Step 1 - Verify setup

cd ~/smart-contract-eval/scripts

python3 01_setup_check.py

Checks that the model is downloaded, Ollama is running, and the dataset files are in place. Fix any reported issues before continuing.

Step 2 - Load the dataset

python3 02_load_dataset.py

Reads relevant_commits.csv and new_fixes.json from the repo and writes them to data/ as JSONL.

Step 3 - Classify vulnerabilities (Task A)

Each commit is assigned one DASP TOP 10 category.

Qwen:

python3 04_classify_vuln.py

DeepSeek:

python3 04_classify_deepseek.py

Output is compared against the paper human labels automatically.

Step 4 - Compare fixes against literature (Task B)

Each commit is judged as aligned or non-aligned with documented academic guidelines.

Qwen:

python3 05_compare_fixes.py

DeepSeek:

python3 05_compare_fixes_deepseek.py

Step 5 - Score novel fixes (Task C)

Novel commits are scored on generalizability, sustainability, and effectiveness (scale 1 to 5).

Qwen:

python3 06_eval_novel_fixes.py

DeepSeek:

python3 06_eval_novel_fixes_deepseek.py

Note: this script requires actual code diffs in the dataset.

Step 6 - Discover novel strategies (Extension)

For non-aligned commits, the LLM checks whether they match any of the 27 known strategies from the paper or represent something genuinely new. New strategies receive a formal written proposal.

Qwen:

python3 07_discover_novel_fixes.py --model qwen

DeepSeek:

python3 07_discover_novel_fixes.py --model deepseek

Step 7 - Statistical analysis

python3 statistical_analysis.py

Performs statistical evaluation and generates summary metrics from the experimental outputs.

Resuming after interruption

All scripts write results line by line and skip already-processed commits on re-run. If a run is interrupted, just run the same command again.

To delete output and start a script from scratch:

python3 07_discover_novel_fixes.py --model qwen --fresh

Output files

python3 final_report.py