Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations

This replication package provides the following files:

File
Description

extraction_evaluation.ipynb
Our evaluation of the data extraction, including code to generate the diagrams shown in the paper.

survey_evaluation.ipynb
Our evaluation of the survey, including code to generate the diagrams shown in the paper.

statistics_evaluation.ipynv
Our statistical evaluation of the survey.

filtering_recall.xlsx
Our evaluation comparing the LLM-based paper filtering to the manual ground truth.

extraction_data_e1_counts.csv
The LLMs used in the papers (extraction question E1), sorted by number of occurrences descending.

extraction_data.csv
The complete dataset from the data extraction.

extraction_round2_data.csv
The complete dataset from the 2nd round of the data extraction (validation round).

llm_strings.csv
A list of DOIs and the corresponding extracted LLMs from the data extraction (used for processing in the evaluation).

llms_extracted_merged.csv
A manual extraction of information from llm_strings.csv, separated into model, version, variant, size. Duplicates were merged, and a count was added. Sorted descending by count.

llms_extracted.csv
A manual extraction of information from llm_strings.csv, separated into model, version, variant, size.

survey_data.csv
The complete dataset from the survey.

survey_questions.csv
The survey questions linked to their IDs and question type.

include_exclude_papers_v3.txt
The final version of our prompt used for paper filtering.

pe_techniques.txt
A list of all extracted PE techniques including the number of occurrences, sorted descending.

survey_questionnaire.pdf
A printout of the online survey we conducted.