Integrating Large Language Models in Software Engineering Education: A Pilot Study through GitHub Repository Mining

Replication Package

This replication package contains all scripts, datasets, and outputs used in the repository mining study on LLMs in Software Engineering Education. It is intended to allow other researchers to reproduce the results, inspect the intermediate data, and extend the analysis.

Contents:

mine_llm_repos.py: Python script used to query the GitHub API, retrieve repositories with relevant keywords, and extract README files and issue discussions. 

analyze_llm_repos.py: Python script for post-processing. It reads the raw dataset, computes summary statistics for motivators and demotivators, and produces CSV summaries.

final-repos_summary.csv: Filtered list of repositories included in the final analysis, with metadata such as repository name, URL, language, stars, forks, and counts of motivator/demotivator hits.

chi_square_results_400_repos.xlsx: Statistical test results, including Chi-square values and p-values for each motivator and demotivator sub-theme across the 400 repositories.

Usage:

Run mine_llm_repos.py to collect repositories and generate the raw CSV.

Run analyze_llm_repos.py to compute theme-level summaries and produce figures.

The .csv and .xlsx files can be opened in Excel or any statistical software for further inspection.

This package ensures transparency of the analysis pipeline and provides a starting point for future work. Researchers may adapt the keyword sets in mine_llm_repos.py to explore other educational contexts or extend the mining to additional platforms such as GitLab, Bitbucket, or SourceForge.