Supplementary Material for "Same Same But Different: Preventing Refactoring Attacks on Software Plagiarism Detection"

README

This repository contains supplementary material for the paper Same Same But Different: Preventing Refactoring Attacks on Software Plagiarism Detection, presented at ICSE 2026 (Research Track).

The paper presents a defense mechanism against automated plagiarism generators that leverages Code Property Graphs and demonstrates its effectiveness against insertion-based and refactoring-based obfuscation attacks. The defense mechanism was also integrated into the software plagiarism detection system JPlag, thereby making it widely accessible.

Purpose

This supplementary material makes the evaluation results described in the aforementioned paper available and reproducible. It contains the executables and data used (as limited by data protection laws), as well as documentation on how to use them.

For our evaluation, we ran a software plagiarism detection pipeline with several configurations on real-world student submissions and on plagiarized submissions generated via several strategies. The underlying plagiarism detection system is JPlag. The evaluation showed that our presented extension NOCTE is successful in defending against various common strategies for obfuscating plagiarism, which state-of-the-art approaches in JPlag were vulnerable to before.

Provenance

This supplementary material can be obtained on Zenodo; the preprint of the paper is available on arXiv.

Content

This supplementary material contains:

code: Contains the executable JAR and Maven project. Contributions are located in JPlag/languages/java-cpg.

transformation/TransformationRepository.java implements the selected transformations.

datasets: Real-world student submissions (datasets/*/human) and automatically generated plagiarized submissions.

gpt: Scripts to generate submissions from task descriptions (gpt/Generated) and from existing submissions (gpt/Obfuscated).

results: Results of our evaluation.

scripts: Scripts to run and analyze the evaluation.

dockerfile: A Dockerfile to facilitate setup.

LICENCE: Licence information.

README.md: This README.

Data

The data consists of submissions for four programming assignments. We provide both real-world student submissions and plagiarized variants generated algorithmically as well as via LLM-based approaches.

The four assignments are boardgame, progpedia19, progpedia56, and tictactoe. Student submissions are located in the subdirectory human.

For data protection reasons, we cannot publish student submissions for boardgame and tictactoe, nor the plagiarized variants derived from them.

The obfuscated variants are provided in these subdirectories:

gpt: Generated by GPT-4 from task descriptions.

gptobf: GPT-4–produced variants of real submissions.

insert: Created by inserting dead statements.

refactor: Produced by an automated refactoring engine.

Dataset provenance:

gpt (link)

gptobf (link)

insert (link)

All data included is open and free. The package is self-contained, and no additional downloads occur. There are no special storage requirements beyond output and temporary files.

Setup

The reproduction process requires:

Hardware

No special requirements. The process runs on consumer-level hardware (Windows/macOS/Linux).

Software

Java JDK 21

Python 3

Note: JDKs other than v21 may cause compatibility problems.

A Dockerfile is provided that installs Ubuntu, required packages, unpacks the supplementary material to /opt/repr/, and runs the evaluation automatically.

Usage

Basic Usage

Verify your Java installation:

java --version

Ensure that JDK 21 (not only JRE) is the default.

Then, from the supplementary material directory:

java -jar ./code/jplag-nocte-and-approaches.jar -h

Look for language modules java and java-cpg (NOCTE) listed at the bottom of the help output.

Manual Replication (See Automatic Workflow below)

To replicate the results, run JPlag (executable and instructions located in ./code/) once for every combination of:

obfuscation approach: insert, refactor, gpt, gptobf

task: BoardGame, Progpedia19, Progpedia56, TicTacToe

approach: base, NOCTE, TSN, SMM, NOCTE+SMM

We now go over each aspect and how to properly configure a JPlag run for it.

Task + Obfuscation approach: JPlag is run on input directories for originals and plagiarized submissions, arranged by task.

Use ./datasets/<task>/human for original submissions.

Use ./datasets/<task>/<subdir> for plagiarized submissions.

Where valid values are:

<task>: BoardGame, Progpedia19, Progpedia56, TicTacToe

<subdir>: insert, refactor, gpt, gptobf

Approach: JPlag flags configure the plagiarism detection variant.

base: -l java

NOCTE: -l java-cpg --normalize

TSN: -l java --normalize

SMM: -l java --match-merging

NOCTE+SMM: -l java-cpg --normalize --match-merging

Clustering is enabled by default but not needed for replication. Disable it with --clustering-skip.

To export the results as CSV files to PWD/<DIR>, use --csv-export -r <DIR>.

Example command (NOCTE, Progpedia19, insertion-based plagiarism):

java -jar ./code/jplag-nocte-and-approaches.jar \
./datasets/Progpedia19/human \
./datasets/Progpedia19/insert \
-l java-cpg --normalize --clustering-skip \
--csv-export -r result-Progpedia19-insert-nocte

Automatic Replication

A provided Dockerfile installs Ubuntu, JDK 21, Python3, downloads and unpacks this supplementary material to /opt/repr/, and sets that directory as the working directory. Running inside Docker avoids heavy filesystem overhead from mounts.

Build and run Docker:

docker build -t repr-nocte .
docker run -it --rm -e INTERACTIVE=false repr-nocte

By default, the container executes the complete evaluation automatically. Set INTERACTIVE=true to run steps manually.

To save JPlag-generated CSVs on the host machine, add:

-v "PATH:/opt/repr/results-csv"

Replacing PATH with a directory on your machine (e.g. $pwd/results-csv; environment syntax varies by OS/shell).

Shell and PowerShell scripts ./scripts/run_all.sh and ./scripts/run_all.ps1 compose and run all valid configurations and collect results in ./results-csv/. Commands are printed for review before execution. These scripts assume the working directory is the archive root.

Inside the prepared Docker container, ./scripts/run_all.sh runs immediately.

Interpretation of the Results

Each CSV in results-csv lists pairs of submissions, the average similarity score between them, and their maximum one-sided similarity. All reported results in the paper use the average similarity.

Plagiarized submission IDs clearly correspond to originals, e.g. insert/plag-subm56 derives from human/subm56.

The script ./scripts/analyze.py aggregates and interprets results. Its output corresponds to Figs. 3–6 in the paper:

insert: Figure 3

refactor: Figure 4

gptobf: Figure 5

gpt: Figure 6

For each configuration, the script classifies pairwise comparisons:

OO — Original to Original

OP — Original to Plagiarized (derived from that original)

all others — discarded

While the original evaluation was performed in R, the Python script yields matching results.