Automatically detecting codeswitching between Nigerian Pidgin English and English

This dataset contains manually annotated ground truth files for evaluating code-switching detection between Nigerian Pidgin English (NPE) and Standard English. It was developed as part of a study evaluating traditional machine learning classifiers, a BiLSTM deep learning model, and compression-based PPM models via the Tawa toolkit.

Two annotation formats are provided:

ml_ground_truth: XML-tagged annotations used to evaluate traditional machine learning and deep learning models, where pidgin and English Segments are marked using XML-style tags.

tawa_ground_truth: Character-level annotations used to evaluate the Tawa PPM compression-based models, where P denotes Nigerian Pidgin English and E denotes Standard English.

All annotations were manually verified by three native speakers of Nigerian Pidgin English. The ground truth was withheld entirely from model training to ensure unbiased evaluation. Full methodological details are provided in the associated publication.

Note: This ground truth dataset is released independently of the full Mixed corpus. The complete corpus cannot be publicly distributed due to copyright restrictions on a subset of the source materials.