DriveMLLM
收藏资源简介：
DriveMLLM数据集由中国科学院自动化研究所等机构创建，专注于自动驾驶场景中的空间理解任务。该数据集包含880张前向摄像头图像，涵盖绝对和相对空间推理任务，并附有丰富的自然语言问题。数据集的创建基于nuScenes数据集，经过严格筛选和标注，确保图像中对象的清晰可见性和空间关系的明确性。DriveMLLM旨在评估和提升多模态大语言模型在自动驾驶中的空间推理能力，解决复杂空间关系理解的问题。
The DriveMLLM dataset, developed by the Institute of Automation of the Chinese Academy of Sciences and other institutions, focuses on spatial understanding tasks in autonomous driving scenarios. This dataset comprises 880 forward-facing camera images, encompasses both absolute and relative spatial reasoning tasks, and is paired with a rich corpus of natural language questions. Built upon the nuScenes dataset, the DriveMLLM dataset has undergone rigorous screening and annotation processes to guarantee clear visibility of objects within the images and unambiguous spatial relationships. DriveMLLM is designed to evaluate and enhance the spatial reasoning capabilities of multimodal large language models in autonomous driving contexts, and to tackle the challenges of comprehending complex spatial relationships.
DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving
数据集概述
- 数据集名称: MLLM_eval_dataset
- 数据来源:
- 图像数据来自nuScenes验证集中的
CAM_FRONT。 - 提供了一个
metadata.jsonl文件，包含图像的属性如location2D。
- 图像数据来自nuScenes验证集中的
- 数据集用途: 用于评估多模态大语言模型在自动驾驶中的空间理解能力。
数据集使用
0. 准备数据集
- 数据集链接: MLLM_eval_dataset
1. 环境设置
- 环境设置文档: Setup Environment
2. 推理
- 推理脚本:
-
GPT API: shell export OPENAI_API_KEY=your_api_key python inference/get_MLLM_output.py --model_type gpt --model gpt-4o --hf_dataset bonbon-rj/MLLM_eval_dataset --prompts_dir prompt/prompts --save_dir inference/mllm_outputs
-
Gemini API: shell export GOOGLE_API_KEY=your_api_key python inference/get_MLLM_output.py --model_type gemini --model models/gemini-1.5-flash --hf_dataset bonbon-rj/MLLM_eval_dataset --prompts_dir prompt/prompts --save_dir inference/mllm_outputs
-
Local LLaVA-Next: shell python inference/get_MLLM_output.py --model_type llava --model lmms-lab/llava-onevision-qwen2-7b-si --hf_dataset bonbon-rj/MLLM_eval_dataset --prompts_dir prompt/prompts --save_dir inference/mllm_outputs
-
Local QWen2-VL: shell python inference/get_MLLM_output.py --model_type qwen --model Qwen/Qwen2-VL-7B-Instruct --hf_dataset bonbon-rj/MLLM_eval_dataset --prompts_dir prompt/prompts --save_dir inference/mllm_outputs
-
3. 评估
- 评估脚本:
-
评估所有结果: shell python evaluation/eval_from_json.py --hf_dataset bonbon-rj/MLLM_eval_dataset --eval_root_dir inference/mllm_outputs --save_dir evaluation/eval_result --eval_model_path all
-
评估特定结果: shell python evaluation/eval_from_json.py --hf_dataset bonbon-rj/MLLM_eval_dataset --eval_root_dir inference/mllm_outputs --save_dir evaluation/eval_result --eval_model_path gemini/gemini-1.5-flash
-
引用
@article{DriveMLLM, title={DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving}, author={Guo, Xianda and Zhang Ruijun and Duan Yiqun and He Yuhang and Zhang, Chenming and Chen, Long}, journal={arXiv preprint arXiv:2411.13112}, year={2024} }
- 1DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving中国科学院自动化研究所 · 2024年
