RT-03 MDE Training Data Text and Annotations
收藏资源简介：
Introduction MDE RT-03 Training Data Text and Annotations corpus was produced by Linguistic Data Consortium (LDC), catalog number LDC2004T12 and ISBN 1-58563-301-1. This data was originally created to support the DARPA EARS (Efficient, Affordable, Reusable Speech-to-Text) Program in Metadata Extraction (MDE). The goal of EARS MDE is to enable technology that can take raw Speech-to-Text output and refine it into forms that are of more use to humans and to downstream automatic processes. The data in this release consists of English Conversational Telephone Speech (CTS) and Broadcast News (BN) transcripts and annotations. The corresponding speech data is available as MDE RT-03 Training Data Speech . Data There are 633 files, totalling approximately 747 MB with a total of 764,978 tokens. The transcripts and annotations cover approximately 20 hours of Broadcast News and over 40 hours of Conversational Telephone Speech data. The annotated data was originally developed to support the DARPA EARS Metadata Extraction (MDE) Program, and was distributed as training data for the RT-03F evaluation cycle. The CTS data was drawn from the Switchboard-1 Release 2 corpus. The BN speech data was drawn from the 1997 English Broadcast News Speech (HUB4) corpus, from four distinct sources: American Broadcasting Company (ABC) (1998, 2001) National Broadcasting Company (NBC) (1998, 2001) Public Radio International (PRI) (1998) Cable News Network (CNN) (2001) Annotations The transcripts within this corpus have been annotated for various kinds of metadata. The goal of MDE is to enable technology that can take raw Speech-To-Text output and refine it into forms that are of more use to humans and to downstream automatic processes. In simple terms, this means the creation of automatic transcripts that are maximally readable. To this end, LDC has defined a SimpleMDE annotation task. Under SimpleMDE, annotators identify four types of fillers: filled pauses like "uh" and "um," discourse markers like "you know," asides and parentheticals, and editing terms like "sorry" and "I mean." Edit disfluencies are also identified; the full extent of the disfluency (or string of adjacent disfluencies) and interruption points are tagged. Annotators further identify SUs (alternately semantic units, sense units, syntactic units, slash units or sentence units); that is, units within the discourse that function to express a complete thought or idea on the part of a speaker. As with disfluency annotation, the goal of SU labeling is to improve transcript readability, here by creating a transcript in which information is presented in small, structured, coherent chunks rather than long turns or stories. There are four types of sentence-level SUs: statements, questions, backchannels and incomplete SUs. To enhance inter-annotator consistency, the annotation task also identifies a number of sub-sentence SU boundaries (coordination and clausal SUs). The docs directory contains the complete set of SimpleMDE annotation guidelines used to create this data. Data Format The data appears in two formats. The AG Atlas (ag.xml) format represents the native annotation format, and utilizes the Annotation Graph Library. This data is best explored using the LDC MDE Toolkit, which is freely available at http://www.ldc.upenn.edu/Projects/MDE/Tools. The data is also provided in RTTM format developed by NIST to support the EARS Program. The RTTM format labels each token in the reference transcript according to the properties it displays: lexeme vs. non-lexeme; edit, filler, SU, etc. Please click here for a RTTM file example. General information about the EARS MDE Annotation effort, including free annotation tools, annotation guidelines and additional information can be found at LDC's EARS MDE Project Page. Updates There are no updates available at this time. Portions (c) 1998 American Broadcasting Company, Inc., (c) 1997-98 Cable News Network, Inc., (c) 1997 Public Radio International, (c) 1997 National Cable Satellite Corporation, (c) 2004 Trustees of the University of Pennsylvania The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.
MDE RT-03训练数据文本与标注语料库由语言数据联盟（Linguistic Data Consortium, LDC）制作，目录编号为LDC2004T12，ISBN为1-58563-301-1。本数据集最初为支持美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）高效经济可复用语音转文本（Efficient, Affordable, Reusable Speech-to-Text, EARS）计划下的元数据提取（Metadata Extraction, MDE）项目而创建。EARS MDE项目的目标是研发相关技术，可将原始语音转文本输出优化为更适配人类使用与下游自动化流程的形式。本发布数据集包含英语会话电话语音（Conversational Telephone Speech, CTS）与广播新闻（Broadcast News, BN）的转录文本及标注文件，对应的语音数据可通过《MDE RT-03训练数据语音（MDE RT-03 Training Data Speech）》获取。 数据部分：本次发布共包含633个文件，总大小约747 MB，总Token数达764,978。该转录文本与标注覆盖约20小时的广播新闻数据，以及超过40小时的会话电话语音数据。本标注数据最初为支持DARPA EARS元数据提取（MDE）项目开发，作为RT-03F评估周期的训练数据发布。会话电话语音数据源自Switchboard-1第二版语料库，广播新闻语音数据则取自1997英语广播新闻语音（HUB4）语料库，涉及4家不同来源：美国广播公司（American Broadcasting Company, ABC，1998、2001）、全国广播公司（National Broadcasting Company, NBC，1998、2001）、国际公共广播（Public Radio International, PRI，1998）以及有线电视新闻网（Cable News Network, CNN，2001）。 标注部分：本语料库中的转录文本已针对各类元数据完成标注。MDE项目的目标是研发相关技术，可将原始语音转文本输出优化为更适配人类使用与下游自动化流程的形式，简言之即生成可读性最大化的自动转录文本。为此，LDC制定了SimpleMDE标注任务规范。在SimpleMDE标注框架下，标注人员需识别四类填充成分：诸如"uh"与"um"的填充停顿、诸如"you know"的话语标记语、插话与插入语，以及诸如"sorry"与"I mean"的修正用语；同时识别编辑式不流畅现象，对不流畅片段（或相邻不流畅序列）的完整范围与中断点进行标记。此外，标注人员需识别语义单元（semantic units, SUs，亦可称为意义单元、句法单元、斜杠单元或句子单元）——即话语中用于完整表达说话者思想或观点的单元。与不流畅标注一致，SU标注的目标同样是提升转录文本的可读性，具体方式是将信息拆分为小型、结构化且连贯的片段，而非以长轮次或完整叙事的形式呈现。句子级SU共分为四类：陈述句、疑问句、反馈语与不完整SU。为提升标注者间一致性，本次标注任务还需识别若干句内SU边界（并列子句与从句语义单元）。docs目录包含了用于制作本数据集的全套SimpleMDE标注指南。 数据格式部分：本数据集提供两种格式。AG图谱（AG Atlas, ag.xml）为原生标注格式，基于标注图库（Annotation Graph Library）构建，最佳浏览方式为使用LDC MDE工具包，该工具包可通过http://www.ldc.upenn.edu/Projects/MDE/Tools免费获取。此外，数据集还提供由美国国家标准与技术研究院（National Institute of Standards and Technology, NIST）为支持EARS计划开发的RTTM格式。RTTM格式会根据参考转录文本中每个Token的属性进行标记，包括词位与非词位、编辑项、填充成分、SU等。如需查看RTTM文件示例，请点击此处。有关EARS MDE标注工作的通用信息，包括免费标注工具、标注指南及其他附加内容，可访问LDC的EARS MDE项目页面获取。 更新情况：目前暂无可用更新。 版权声明：部分内容版权所有：(c) 1998 美国广播公司股份有限公司，(c) 1997-1998 有线电视新闻网股份有限公司，(c) 1997 国际公共广播，(c) 1997 国家有线电视卫星公司，(c) 2004 宾夕法尼亚大学托管方。《The World》为国际公共广播与英国广播公司（British Broadcasting Corporation, BBC）联合制作，由波士顿WGBH工作室出品。
