Multi-Human Interactive Talking Dataset

Existing studies on talking video generation have predominantly focused on single-person monologues or isolated facial animations, limiting their applicability to realistic multi-human interactions. To bridge this gap, we introduce MIT, a large-scale dataset specifically designed for multi-human talking video generation. To this end, we develop an automatic pipeline that collects and annotates multi-person conversational videos. The resulting dataset comprises 12 hours of high-resolution footage, each featuring two to four speakers, with fine-grained annotations of body poses and speech interactions. It captures natural conversational dynamics in multi-speaker scenario, offering a rich resource for studying interactive visual behaviors. To demonstrate the potential of MIT, we furthur propose CovOG, a baseline model for this novel task. It integrates a Multi-Human Pose Encoder (MPE) to handle varying numbers of speakers by aggregating individual pose embeddings, and an Interactive Audio Driver (IAD) to modulate head dynamics based on speaker-specific audio features. Together, these components showcase the feasibility and challenges of generating realistic multi-human talking videos, establishing MIT as a valuable benchmark for future research. The dataset and code will be public available.

现有对话视频生成领域的研究大多聚焦于单人独白或孤立的面部动画生成，这极大限制了其在真实多人类交互场景中的适用性。为填补这一研究空白，我们提出了MIT，一款专为多人类对话视频生成任务设计的大规模数据集。为此，我们搭建了一套自动化采集与标注流程，用于获取多人对话视频数据。最终构建的数据集包含12小时的高分辨率视频素材，每段视频均涵盖2至4位对话者，并附带精细的身体姿态与语音交互标注信息。该数据集能够捕捉多说话者场景下的自然对话动态，为交互式视觉行为研究提供了丰富的资源支撑。为验证MIT的应用潜力，我们进一步提出了CovOG，一款针对该全新任务的基准模型。该模型集成了多人姿态编码器（Multi-Human Pose Encoder，MPE），通过聚合单一个体的姿态嵌入特征来适配不同数量的对话者；同时引入了交互式音频驱动模块（Interactive Audio Driver，IAD），可基于针对特定说话者的音频特征调节头部动作动态。上述两大模块共同展示了生成真实感多人类对话视频的可行性与现存挑战，同时确立了MIT作为未来相关研究的优质基准数据集的地位。该数据集与配套代码将对外开源发布。