khang123452/GR1-Manipulation-IsaacSim-Rendered

--- license: cc-by-4.0 task_categories: - robotics - video-classification tags: - robotics - manipulation - sim-to-real - cosmos-transfer - isaac-sim - gr1-humanoid - pick-and-place pretty_name: GR1 Manipulation - Isaac Sim Rendered (RGB/Depth/Seg) size_categories: - 10K<n<100K --- # GR1 Manipulation — Isaac Sim Rendered Dataset ## Dataset Description Multi-modal rendered dataset of **NVIDIA GR1 humanoid robot** performing tabletop manipulation tasks in Isaac Sim. Each episode is rendered as 3 synchronized MP4 videos: - **RGB** — ray-traced color video - **Depth** — normalized depth map (grayscale, 0–10m range) - **Segmentation** — semantic segmentation with per-object class coloring This dataset is designed as input for **NVIDIA Cosmos Transfer 2.5** to generate photorealistic sim-to-real augmented training data for robot learning. ## Dataset Structure ``` {task_name}/ episode_NNNN_rgb.mp4 # 1280x720, 20fps, RayTracedLighting episode_NNNN_depth.mp4 # 1280x720, 20fps, normalized depth episode_NNNN_seg.mp4 # 1280x720, 20fps, semantic segmentation ``` ## Tasks | # | Task Name | Episodes | Frames/Episode | |---|-----------|----------|----------------| | 1 | `PnPBottleToCabinetClose` | 458 | ~120–400 | | 2 | `PnPCanToDrawerClose` | 2 | ~120–400 | | 3 | `PnPCupToDrawerClose` | 2 | ~120–400 | | 4 | `PnPMilkToMicrowaveClose` | 2 | ~120–400 | | 5 | `PnPPotatoToMicrowaveClose` | 2 | ~120–400 | | 6 | `PnPWineToCabinetClose` | 2 | ~120–400 | | 7 | `PosttrainPnPNovelFromCuttingboardToBasketSplitA` | 2 | ~120–400 | | 8 | `PosttrainPnPNovelFromCuttingboardToCardboardboxSplitA` | 2 | ~120–400 | | 9 | `PosttrainPnPNovelFromCuttingboardToPanSplitA` | 2 | ~120–400 | | 10 | `PosttrainPnPNovelFromCuttingboardToPotSplitA` | 2 | ~120–400 | | 11 | `PosttrainPnPNovelFromCuttingboardToTieredbasketSplitA` | 2 | ~120–400 | | 12 | `PosttrainPnPNovelFromPlacematToBasketSplitA` | 2 | ~120–400 | | 13 | `PosttrainPnPNovelFromPlacematToBowlSplitA` | 1 | ~120–400 | | 14 | `PosttrainPnPNovelFromPlacematToPlateSplitA` | 2 | ~120–400 | | 15 | `PosttrainPnPNovelFromPlacematToTieredshelfSplitA` | 2 | ~120–400 | | 16 | `PosttrainPnPNovelFromPlateToBowlSplitA` | 2 | ~120–400 | | 17 | `PosttrainPnPNovelFromPlateToCardboardboxSplitA` | 1 | ~120–400 | | 18 | `PosttrainPnPNovelFromPlateToPanSplitA` | 2 | ~120–400 | | 19 | `PosttrainPnPNovelFromPlateToPlateSplitA` | 1 | ~120–400 | | 20 | `PosttrainPnPNovelFromTrayToCardboardboxSplitA` | 2 | ~120–400 | | 21 | `PosttrainPnPNovelFromTrayToPlateSplitA` | 2 | ~120–400 | | 22 | `PosttrainPnPNovelFromTrayToPotSplitA` | 2 | ~120–400 | | 23 | `PosttrainPnPNovelFromTrayToTieredbasketSplitA` | 1 | ~120–400 | | 24 | `PosttrainPnPNovelFromTrayToTieredshelfSplitA` | 2 | ~120–400 | ## Recording Settings | Setting | Value | |---------|-------| | **Camera** | `egoview` (1st-person, attached to `robot0_head_pitch`) | | **FOV** | 90° vertical | | **Resolution** | 1280×720 | | **FPS** | 20 | | **Renderer** | Isaac Sim RayTracedLighting, SPP=32 | | **Lighting** | Dome light, intensity=1500 | | **USD Export** | robosuite exporter, geomgroup=[0,1,0,0,0,0] (visual meshes only) | ## Source Data - **Robot**: NVIDIA GR1 humanoid (from gr00trobosuite) - **Tasks**: RoboCasa GR1 tabletop pick-and-place tasks - **Simulation**: MuJoCo → USD → Isaac Sim Replicator pipeline - **Original trajectories**: 1000 episodes per task in HDF5 format ## Intended Use Input for Cosmos Transfer 2.5 world generation model to produce photorealistic augmented manipulation videos for robot policy training (sim-to-real transfer). ## License CC-BY-4.0许可证：CC BY 4.0 任务类别： - 机器人学 - 视频分类 标签： - 机器人学 - 操作 - 仿真到真实（sim-to-real） - cosmos-transfer - isaac-sim - gr1-humanoid - 拾放任务 可视化名称：GR1 操作任务——Isaac Sim 渲染数据集（RGB/深度/分割） 样本规模：10000 < 样本数 < 100000 # GR1 操作任务——Isaac Sim 渲染数据集 ## 数据集说明 本数据集为多模态渲染数据集，基于**NVIDIA GR1人形机器人（NVIDIA GR1 humanoid robot）**在Isaac Sim中完成桌面操作任务的场景构建。每个任务片段均渲染为三路同步MP4视频： - **RGB**：光线追踪色彩视频 - **深度（Depth）**：归一化深度图（灰度格式，取值范围0~10米） - **分割（Segmentation）**：逐目标类别着色的语义分割视频 本数据集专为**NVIDIA Cosmos Transfer 2.5**设计，用于生成面向机器人学习的高逼真度仿真到真实（sim-to-real）增强训练数据。 ## 数据集结构 {task_name}/ episode_NNNN_rgb.mp4 # 分辨率1280×720，帧率20fps，采用光线追踪照明 episode_NNNN_depth.mp4 # 分辨率1280×720，帧率20fps，归一化深度图 episode_NNNN_seg.mp4 # 分辨率1280×720，帧率20fps，语义分割视频 ## 任务列表 | 序号 | 任务名称 | 片段数 | 每片段帧数 | |---|-----------|----------|----------------| | 1 | `PnPBottleToCabinetClose` | 458 | ~120–400 | | 2 | `PnPCanToDrawerClose` | 2 | ~120–400 | | 3 | `PnPCupToDrawerClose` | 2 | ~120–400 | | 4 | `PnPMilkToMicrowaveClose` | 2 | ~120–400 | | 5 | `PnPPotatoToMicrowaveClose` | 2 | ~120–400 | | 6 | `PnPWineToCabinetClose` | 2 | ~120–400 | | 7 | `PosttrainPnPNovelFromCuttingboardToBasketSplitA` | 2 | ~120–400 | | 8 | `PosttrainPnPNovelFromCuttingboardToCardboardboxSplitA` | 2 | ~120–400 | | 9 | `PosttrainPnPNovelFromCuttingboardToPanSplitA` | 2 | ~120–400 | | 10 | `PosttrainPnPNovelFromCuttingboardToPotSplitA` | 2 | ~120–400 | | 11 | `PosttrainPnPNovelFromCuttingboardToTieredbasketSplitA` | 2 | ~120–400 | | 12 | `PosttrainPnPNovelFromPlacematToBasketSplitA` | 2 | ~120–400 | | 13 | `PosttrainPnPNovelFromPlacematToBowlSplitA` | 1 | ~120–400 | | 14 | `PosttrainPnPNovelFromPlacematToPlateSplitA` | 2 | ~120–400 | | 15 | `PosttrainPnPNovelFromPlacematToTieredshelfSplitA` | 2 | ~120–400 | | 16 | `PosttrainPnPNovelFromPlateToBowlSplitA` | 2 | ~120–400 | | 17 | `PosttrainPnPNovelFromPlateToCardboardboxSplitA` | 1 | ~120–400 | | 18 | `PosttrainPnPNovelFromPlateToPanSplitA` | 2 | ~120–400 | | 19 | `PosttrainPnPNovelFromPlateToPlateSplitA` | 1 | ~120–400 | | 20 | `PosttrainPnPNovelFromTrayToCardboardboxSplitA` | 2 | ~120–400 | | 21 | `PosttrainPnPNovelFromTrayToPlateSplitA` | 2 | ~120–400 | | 22 | `PosttrainPnPNovelFromTrayToPotSplitA` | 2 | ~120–400 | | 23 | `PosttrainPnPNovelFromTrayToTieredbasketSplitA` | 1 | ~120–400 | | 24 | `PosttrainPnPNovelFromTrayToTieredshelfSplitA` | 2 | ~120–400 | ## 录制设置 | 设置项 | 参数值 | |---------|-------| | **相机** | `egoview`（第一人称视角，挂载于`robot0_head_pitch`节点） | | **视场角** | 90°垂直视场 | | **分辨率** | 1280×720 | | **帧率** | 20fps | | **渲染器** | Isaac Sim 光线追踪照明，采样数（SPP）=32 | | **照明方案** | 穹顶光源，强度=1500 | | **USD导出** | robosuite导出器，geomgroup=[0,1,0,0,0,0]（仅导出视觉网格） | ## 源数据 - **机器人平台**：NVIDIA GR1人形机器人（源自gr00trobosuite项目） - **任务集**：RoboCasa GR1桌面拾放任务 - **仿真流程**：MuJoCo → USD → Isaac Sim Replicator - **原始轨迹**：每个任务含1000个片段，格式为HDF5 ## 预期用途 本数据集可作为Cosmos Transfer 2.5场景生成模型的输入，用于生成高逼真度的增强型操作视频，以支撑机器人策略训练（仿真到真实（sim-to-real）迁移）。 ## 许可证 CC BY 4.0