MMGR

# MMGR-Open This repository is the public release mirror for the MMGR benchmark assets that are currently organized under `code/` and `data/`. ## Layout ```text code/ Evaluation prompts and scripts. data/ Input images, generation prompts, ground-truth solutions where applicable. ``` The machine-readable release index is `dataset_manifest.json`. ## Dataset Manifest `dataset_manifest.json` describes: - task names and directory locations; - sample counts for prompts, images, and solutions; - how images, prompts, solutions, and task-level metadata are paired; - which missing fields are intentional and which still require source mapping. Use the manifest as the authoritative starting point for consuming this release. ## Pairing Rules ### Abstract Reasoning ARC, Maze, and Sudoku provide one input image per ground-truth solution image. Their image and solution files are paired by matching task-specific filenames or relative directory structure. Math uses rendered problem images plus aggregated JSON files under `data/solution/abstract_reasoning/math/`. The JSON files contain the corresponding problem statements and solutions. ### Embodied Navigation Embodied-navigation prompt templates live in `data/prompt/embodied_nav/<task>/`. For color-goal samples, the visible target is encoded in the input image and the matching floor/quality/turn subset directory. For object-goal samples, prompts contain `{LOCATION_DESCRIPTION}`. Resolve that placeholder with: ```text data/prompt/embodied_nav/location_descriptions_manifest.jsonl ``` Each JSONL record maps one public object-goal input image to its `location_description`, prompt template paths, subset directory, and source file. The original source text files are not required for normal use; if retained later, they should be treated only as optional human-readable copies under `data/prompt/embodied_nav/location_descriptions/<task>/<subset_dir>/<sample_id>.txt`. The location-description manifest follows the canonical benchmark split: 60 object-goal records per embodied-navigation task, 240 records total. It does not include fallback records from another quality or subset. Embodied-navigation images also follow the canonical benchmark split: each of the four tasks has 60 color-goal input images and 60 object-goal input images, for 480 navigation images total. Navigation ground-truth solution files are not included in this release mirror yet. Their source mapping is still pending. ### Physical Commonsense Physical commonsense is a prompt-driven video-generation/evaluation task. It does not require input images or solution images in this release layout. The prompt JSON files under `data/prompt/physical_commonsense/` define the generation prompts; the VLM evaluation prompt is under `code/physical_commonsense/vlm_based_evaluation/`. `sports_prompts.json` already contains `prompt`, `physics_focus`, and `expected_motion`. `physical_concept_prompts.json` still needs an explicit mapping for the VLM placeholders `prompt_text`, `physics_info`, and `expected_motion`. ## Known Missing Or Pending Items - `data/metadata/` is not present. The root `dataset_manifest.json` currently serves as the release-level manifest, but task-level metadata may still need to be added. - Embodied-navigation solutions are not present and still require an approved mapping from source ground-truth files. - Physical commonsense images and solutions are intentionally absent for the current prompt-only video task design. - Two duplicate Google Drive math JSON files were observed during local sync; the local mirror collapsed them to unique files.