ISL-CSLTR: Indian Sign Language Dataset for Continuous Sign Language Translation and Recognition

Sign language is a cardinal element for communication between deaf and dumb community. Sign language has its own grammatical structure and gesticulation nature. Research on SLRT focuses a lot of attention in gesture identification. Sign language comprises of manual gestures performed by hand poses and non-manual features expressed through eye, mouth and gaze movements. The sentence-level completely labelled Indian Sign Language dataset for Sign Language Translation and Recognition (SLTR) research is developed. The ISL-CSLTR dataset assists the research community to explore intuitive insights and to build the SLTR framework for establishing communication with the deaf and dumb community using advanced deep learning and computer vision methods for SLTR purposes. This ISL-CSLTR dataset aims in contributing to the sentence level dataset created with two native signers from Navajeevan, Residential School for the Deaf, College of Spl. D.Ed & B.Ed, Vocational Centre, and Child Care & Learning Centre, Ayyalurimetta, Andhra Pradesh, India and four student volunteers from SASTRA Deemed University, Thanjavur, Tamilnadu. The ISL-CSLTR corpus consists of a large vocabulary of 700 fully annotated videos, 18863 Sentence level frames, and 1036 word level images for 100 Spoken language Sentences performed by 7 different Signers. This corpus is arranged based on signer variants and time boundaries with fully annotated details and it is made available publicly. The main objective of creating this sentence level ISL-CSLRT corpus is to explore more research outcomes in the area of SLTR. This completely labelled video corpus assists the researchers to build framework for converting spoken language sentences into sign language and vice versa. This corpus has been created to address the various challenges faced by the researchers in SLRT and significantly improves translation and recognition performance. The videos are annotated with relevant spoken language sentences provide clear and easy understanding of the corpus data. Acknowledgements: The research was funded by the Science and Engineering Research Board (SERB), India under Start-up Research Grant (SRG)/2019–2021 (Grant no. SRG/2019/001338). And also, we thank all the signers for their contribution in collecting the sign videos and the successful completion of the ISL-CSLTR corpus. We would like to thank Navajeevan, Residential School for the Deaf, College of Spl. D.Ed & B.Ed, Vocational Centre, and Child Care & Learning Centre, Ayyalurimetta, Andhra Pradesh, India for their support and contribution.

手语是聋哑人群体间沟通的核心要素。手语拥有独立的语法结构与手势表达的固有属性。手语翻译与识别（Sign Language Translation and Recognition，SLRT）领域的研究对手势识别给予了大量关注。手语由手部姿态形成的手动手势，以及通过眼部、唇部与视线运动传递的非手动特征共同构成。
本研究构建了适用于SLRT研究的句子级全标注印度手语（Indian Sign Language，ISL）数据集。印度手语-SLRT语料库（Indian Sign Language-Corpus for SLTR，ISL-CSLTR）可助力研究人员探索直观研究视角，并基于先进深度学习与计算机视觉方法搭建SLRT框架，以实现与聋哑人群体的有效沟通。
本ISL-CSLTR数据集旨在完善句子级数据集的构建，数据采集自印度安得拉邦阿亚卢里梅塔的Navajeevan聋哑寄宿学校、特殊教育文凭与教育学士学院、职业中心及儿童护理与学习中心的两名本土手语使用者，以及印度泰米尔纳德邦坦贾武尔的萨斯特拉Deemed大学的四名学生志愿者。
ISL-CSLTR语料库包含7名不同手语使用者完成的100条口语语句对应的700段全标注视频、18863个句子级帧以及1036张单词级图像，涵盖丰富的词汇量。该语料库依据手语使用者差异与时间边界进行整理，并附带完整的标注细节，且已公开可用。
构建该句子级ISL-CSLTR语料库的核心目标，是在SLRT领域探索更多研究成果。这款全标注视频语料库可助力研究人员搭建将口语语句转换为手语，以及将手语转换为口语语句的框架。本语料库的构建旨在解决SLRT领域研究人员面临的各类挑战，并可显著提升翻译与识别性能。视频均配有对应的口语语句标注，便于研究人员清晰理解语料库数据。
致谢
本研究获印度科学与工程研究委员会（Science and Engineering Research Board，SERB）启动研究基金（SRG/2019–2021，项目编号：SRG/2019/001338）资助。同时，我们感谢所有手语参与者为手语视频采集以及ISL-CSLTR语料库的顺利完成所做出的贡献。我们还要感谢印度安得拉邦阿亚卢里梅塔的Navajeevan聋哑寄宿学校、特殊教育文凭与教育学士学院、职业中心及儿童护理与学习中心提供的支持与帮助。