PRIME-Py: A Large-Scale Python Function Dataset

PRIME-Py is a large-scale dataset of 1,997,535 Python functions extracted from 2,797 open-source GitHub repositories. Each function is described through two types of features:

Rule-based features: structural metrics extracted by Lizard (lines of code, cyclomatic complexity, parameter count, token count) and lexical features extracted by Python’s AST module (function call counts, variable count, branch count, class membership).

Neural summaries: natural language descriptions of each function’s behaviour generated by the SEBIS CodeTrans T5 model, with 99.2% coverage.

Repositories were collected from GitHub between 2023 and 2024 using the following criteria: Python as the primary language, at least 50 stars, at least 500 lines of Python, and non-forked. The dataset is split at the project level (80/10/10, seed=42) to prevent data leakage between training and evaluation sets.

Files

master_dataset.parquet (1,997,535 rows)  Base dataset with identity information and structural metrics from Lizard.

master_dataset_with_lexical.parquet (1,997,535 rows)  Adds lexical features extracted by Python’s AST module.

master_dataset_with_codet5.parquet (1,997,535 rows)  Adds neural code summaries from SEBIS CodeTrans T5. This is the complete feature dataset.

train.parquet (1,554,896 rows, 2,237 projects)  Training split of the complete feature dataset.

val.parquet (165,226 rows, 280 projects)  Validation split of the complete feature dataset.

test.parquet (277,413 rows, 280 projects)  Test split of the complete feature dataset.

Related Resources

Replication package (scripts, annotation materials):  https://github.com/[yourusername]/PRIME-Py

Related research article:  Alehaidib R, Ghoneim A, Alrashoud M. 2026. Large-Scale Empirical Study of Code Smell and Anti-Pattern Detection in Python Open-Source Software. PeerJ Computer Science. DOI: [to be added on acceptance]

Authors

Reem Alehaidib, Ahmed Ghoneim, Mubarak Alrashoud

Department of Software Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia.

Licence

Creative Commons Attribution 4.0 International (CC BY 4.0).