nygdon/vimqa-generated-answers-pass1

---
license: other
language:
- vi
task_categories:
- question-answering
- multiple-choice
size_categories:
- 1K<n<10K

configs:
- config_name: agree_set
data_files: "evaluation_results/agree.jsonl"
- config_name: majority_set
data_files: "evaluation_results/majority.jsonl"
- config_name: conflict_set
data_files: "evaluation_results/conflict.jsonl"
- config_name: gemma_output
data_files:
- split: test
path: "raw_outputs/results_pass1_gemma.jsonl"
- config_name: llama_output
data_files:
- split: test
path: "raw_outputs/results_pass1_llama.jsonl"
- config_name: qwen_output
data_files:
- split: test
path: "raw_outputs/results_pass1_qwen.jsonl"
---
# Vi-MQA - Pass 1 Generated Answers & Evaluation

This repo contains the Pass 1 outputs and evaluation results for the **Vi-MQA Dataset** from the [VMLU Benchmark Suite](https://vmlu.ai/) with a total of 4,762 records.

## Folder Structure

### 1. Model Outputs (`raw_outputs/`)

Contains the formatted outputs from the 3 models evaluated in Pass 1:

- `results_pass1_gemma.jsonl` (Gemma 4 31B IT)
- `results_pass1_llama.jsonl` (Llama 4 Scout)
- `results_pass1_qwen.jsonl` (Qwen3 32B)

### 2. Evaluation Results (`evaluation_results/`)

Contains the evaluation results comparing the outputs of 3 models:

- **`agree.jsonl`** (1,152 samples): High confidence. All 3 models give the exact same answer.
- **`majority.jsonl`** (381 samples): Good reliability. 2 out of 3 models agreed.
- **`conflict.jsonl`** (46 samples): No consensus. **Requires Pass 2 and Manual Review.**

### Scripts

- `evaluate.py`: The evaluation script that calculates agreement, majority and conflict sets.