alyah-emirati-benchmark_verified

# Alyah Emirati Benchmark (verified) A filtered version of **Alyah**, the Emirati-dialect multiple-choice benchmark from TII, with samples flagged by a multi-judge audit removed. > **What "verified" means.** Samples were removed when all three LLM judges flagged a significant defect. This is not a human review, and only samples that baseline models answered incorrectly were audited, so some defects may remain in the kept data. **This repository does not contain the Alyah questions.** The source dataset is public but states no license, so we publish the filter instead: which source rows to keep, plus a script that applies it to the source. ## How to use ```bash pip install pandas pyarrow huggingface_hub python apply_filter.py alyah_verified.parquet ``` [`apply_filter.py`](apply_filter.py) downloads the source at the revision below, checks every row against the fingerprints in `ids.csv`, and writes the 1,158-row verified test set. It stops with an error if the source has changed. Or filter it yourself: ```python import pandas as pd ids = pd.read_csv("hf://datasets/IFM/alyah-emirati-benchmark_verified/ids.csv") source = pd.read_parquet( "hf://datasets/tiiuae/alyah-emirati-benchmark@e404e3e7745f3401d3c13eb2e97be36bff898b77/data/test-00000-of-00001.parquet" ) verified = source[ids["status"].eq("kept").to_numpy()].reset_index(drop=True) # 1,158 rows ``` ## Files - `ids.csv`: one row per source row (1,173 rows): - `row`: the 0-based row index in the source file. - `category`: the source `category`. - `status`: `kept` (1,158) or `removed` (15). - `fingerprint`: the first 16 hex characters of a SHA-256 over the row's `query`, `option_1`–`option_4`, `correct_answer` and `category`, used to detect changes in the source. - `apply_filter.py`: rebuilds the verified set from the source. ## Source - Original dataset: [`tiiuae/alyah-emirati-benchmark`](https://huggingface.co/datasets/tiiuae/alyah-emirati-benchmark), revision `e404e3e7745f3401d3c13eb2e97be36bff898b77`, file `data/test-00000-of-00001.parquet` (1,173 rows). The data file is unchanged since the upload commit `134f969e`; later commits only edit the README. - Columns of the rebuilt set, as in the source: `query`, `option_1`–`option_4`, `correct_answer` (the correct option number, 1–4) and `category`. ## Audit and filtering Samples were screened with the multi-judge audit described in [`IFM/discarded_samples`](https://huggingface.co/datasets/IFM/discarded_samples) (config `Alyah (Emirati)`). Three language-model judges (GPT-5.6 Sol, Gemini 3.8 Flash, Claude Opus 5) assessed candidate samples that strong baselines answered incorrectly. A sample is excluded only when **all three** judges report a *significant* issue (a problem with the question, the options or the gold label that prevents reliable answering or scoring). Each discarded record's `doc_id` is the 0-based position of the sample within its category in the source file; `ids.csv` gives the matching source row (`row`) for each excluded sample. All other rows are kept unchanged and in their original order. | Category | Original rows | Excluded | Kept | |---|---:|---:|---:| | Language & Dialect | 619 | 10 | 609 | | Imagery & Figurative Meaning | 121 | 1 | 120 | | Etiquette & Values | 173 | 0 | 173 | | Religious & Social Sensitivity | 78 | 0 | 78 | | Historical & Heritage Knowledge | 89 | 0 | 89 | | Greetings & Daily Expressions | 61 | 4 | 57 | | Poetry & Creative Expression | 32 | 0 | 32 | | **Total** | **1,173** | **15** | **1,158** | ## License The source dataset states no license, so this repository redistributes none of its text. The files here (`ids.csv`, `apply_filter.py` and this card) are released under CC BY 4.0. The verified set you build from the source remains subject to the source's terms. ## Citation Please cite the original authors: ```bibtex @misc{emirati_dialect_benchmark_2026, title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models}, author = {Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid}, year = {2026}, month = {january}, url = {https://huggingface.co/datasets/tiiuae/alyah-emirati-benchmark} } @misc{jais3_2026, title = {Jais 3: Introducing Arabic-centric Agentic AI}, author = {{IFM Team, Inception42 Team, and Cerebras Team}}, year = {2026}, url = {https://ifm.ai/blog/jais3/}, } ```