LatentGUIWorld

# LatentGUIWorld LatentGUIWorld is the interactive GUI benchmark introduced in **Learn from Move: the Next Step for GUI Agents**. It contains **900 test episodes** across six environments, with **150 episodes per environment**, and **24,000 training trajectories** yielding **154,713 SFT examples**. Observations use a **1280 × 720** viewport. [Code and environment runtime](https://github.com/ti-mm/LearnFromMove) ## Environments | Configuration | Task | Episodes | |---|---|---:| | `drag_egocentric` | Egocentric Drag | 150 | | `drag_exocentric` | Exocentric Drag | 150 | | `rotation_inner` | Inner Rotation | 150 | | `rotation_outer` | Outer Rotation | 150 | | `ten_choice_egocentric` | Egocentric Ten-Choice | 150 | | `ten_choice_exocentric` | Exocentric Ten-Choice | 150 | Drag places a colored shape into a matching outline. Ten-Choice identifies a target among ten candidates through hover-revealed text. Rotation uses a horizontal slider to align the inner or outer image region. Agents use interaction feedback to adapt their actions to each environment's dynamics. The two environments in each task family share 150 paired scene identities, giving 450 scene pairs and 900 episodes. The `pair_id` identifies the shared scene; the `episode_id` identifies a particular environment episode. The six environment configurations expose a `test` split. The default `all` configuration combines these subsets into 900 test rows. The `sft` configuration exposes the supervised examples through a `train` split. The `distribution` field identifies the 75 IID and 75 OOD episodes in each Drag environment. Rotation and Ten-Choice each use fixed test sets without an IID/OOD subdivision, so their `distribution` values are `null`. The held-out factors follow the paper's test-set construction. Drag's IID episodes use training shape categories, while its OOD episodes use held-out shape categories. All Ten-Choice test scenes draw from 240 messages disjoint from the 80 training messages. Rotation uses 150 test background images disjoint from the 1,000 training backgrounds. Ten-Choice and Rotation therefore evaluate held-out content across their full test sets. ## Load test episode records ```python import json from datasets import load_dataset repo_id = "OpenMOSS-Team/LatentGUIWorld" episodes = load_dataset(repo_id, "all", split="test") drag = load_dataset(repo_id, "drag_egocentric", split="test") episode = json.loads(drag[0]["episode_json"]) ``` | Field | Meaning | |---|---| | `episode_id`, `pair_id` | Episode and paired-scene identifiers | | `suite_id`, `variant`, `family` | Benchmark suite, environment, and task family | | `instruction` | Task instruction | | `exploration_level` | Exploration category | | `distribution` | `iid` or `ood` for Drag; `null` for Rotation and Ten-Choice | | `canonical_case_path` | Scene metadata path relative to the dataset root | | `episode_json` | Complete runtime episode configuration serialized as JSON | The runtime consumes the full configuration, including hidden dynamics and success criteria. Agent observations consist of task instructions, screenshots, and the structured metadata selected by the runtime's observation contract. ## Training data The training set contains 4,000 trajectories per environment. Each SFT example uses up to three recent screenshots and preceding assistant responses to predict one Think/action response. | Environment | Trajectories | SFT examples | |---|---:|---:| | Egocentric Drag | 4,000 | 24,452 | | Exocentric Drag | 4,000 | 16,000 | | Inner Rotation | 4,000 | 28,479 | | Outer Rotation | 4,000 | 28,349 | | Egocentric Ten-Choice | 4,000 | 31,169 | | Exocentric Ten-Choice | 4,000 | 26,264 | | **Total** | **24,000** | **154,713** | ```python from datasets import load_dataset train = load_dataset("OpenMOSS-Team/LatentGUIWorld", "sft", split="train") ``` Training rows contain `messages`, `images`, `tools`, and `metadata`. `messages` contains the input conversation and target assistant response. `images` contains dataset-relative paths and the image pixel limit. `metadata` identifies the trajectory, environment, action index, and row index. The target consists of a Think rationale followed by a JSON action. Images are rendered by the accompanying environment runtime and stored as lossless WebP files in tar archives (47.0 GiB). Install the GitHub package and prepare the training files with: ```bash pip install -e '.[data]' latentguiworld-prepare-training --download --dataset-root data/LatentGUIWorld \ --output data/training/benchmark/train.parquet ``` Run these commands from the code repository. The preparation command downloads and extracts the image archives and resolves local image paths in the output Parquet. Pass that file to the `benchmark_sft` training recipe. `training/trajectories.jsonl.gz` provides complete trajectories with task instructions, ordered observations, and assistant responses. The final observation follows the final action. `training/dataset.json` indexes the SFT shards and image archives. ## Environment interaction This dataset contains scene configurations, rendering assets, and the Ten-Choice HTML/JavaScript scenes. The complete environment runtime, mouse-action interface, and evaluation code are available in the [GitHub repository](https://github.com/ti-mm/LearnFromMove). ## Download and run the benchmark ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="OpenMOSS-Team/LatentGUIWorld", repo_type="dataset", local_dir="benchmark", ignore_patterns=["training/**"], ) ``` Install the latest environment runtime from the [code repository](https://github.com/ti-mm/LearnFromMove), then pass the downloaded manifest to the environment or evaluation entry point: ```bash python -m six_environments list --manifest benchmark/manifest.json python -m six_environments serve --manifest benchmark/manifest.json --port 8765 latentguiworld-eval --manifest benchmark/manifest.json \ --model models/latentlearner --output results/latentlearner ``` Paths are relative to the directory where the commands are run. The evaluator scores the first release attempt and reports success rates per environment. ## Files ```text README.md LICENSE NOTICE.md manifest.json # Benchmark runtime manifest: 900 episodes data/*.jsonl # Six Hugging Face subsets: 150 rows each cases/drag/*/meta.json cases/rotation/*/meta.json cases/ten_choice/*/meta.json cases/ten_choice/*/index.html cases/ten_choice/*/assets/icon.svg assets/rotation/*.jpg training/data/*.jsonl.gz # Hugging Face SFT rows training/data/*.parquet # Training Parquet shards training/archives/*.tar # Lossless observation images training/trajectories.jsonl.gz # Complete demonstration trajectories training/dataset.json # Training data index training/attribution.jsonl # Training background source references ``` The JSONL files provide a browsable view of the episodes in `manifest.json`. Scene paths in both representations resolve against the dataset root. ## Attribution The original release license is included in [LICENSE](LICENSE). Rotation backgrounds originate from Open Images and, for 42 training backgrounds, The Metropolitan Museum of Art. Source references are included in scene metadata and `training/attribution.jsonl`. Image rights remain with their respective owners. See [NOTICE.md](NOTICE.md).