GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2

Name: GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2
Creator: Linguistic Data Consortium
Published: 2020-11-30T15:50:05+08:00

DataCite Commons2021-07-01 更新2025-04-16 收录

机器翻译

阿拉伯语处理

数据链接：

https://catalog.ldc.upenn.edu/LDC2008T09 数据链接链接失效反馈

官方服务：

资源简介：

<h3>Introduction</h3> GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2 is the second of the three-part GALE Phase 1 Arabic Broadcast News Parallel Text, which, along with other corpora, was used as training data in year 1 (Phase 1) of the DARPA-funded GALE program. <a href="http://catalog.ldc.upenn.edu/LDC2007T24" rel="nofollow">GALE Phase 1 Arabic Broadcast News Parallel Text - Part 1</a> was released in 2007. GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2 contains transcripts and English translations of 10.7 hours of Arabic broadcast news programming selected from various sources. This corpus does not contain the audio files from which the transcripts and translations were generated. LDC has released the following GALE Phase 1 & 2 Arabic Parallel Text data sets: <ul> <li>GALE Phase 1 Arabic Broadcast News Parallel Text - Part 1 (<a href="../../../LDC2007T24">LDC2007T24</a>)</li> <li>GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2 (<a href="../../../LDC2008T09">LDC2008T09</a>)</li> <li>GALE Phase 1 Arabic Blog Parallel Text (<a href="../../../LDC2008T02">LDC2008T02</a>)</li> <li>GALE Phase 1 Arabic Newsgroup Parallel Text - Part 1 (<a href="../../../LDC2009T03">LDC2009T03</a>)</li> <li>GALE Phase 1 Arabic Newsgroup Parallel Text - Part 2 (<a href="../../../LDC2009T09">LDC2009T09</a>)</li> <li>GALE Phase 2 Arabic Broadcast Conversation Parallel Text Part 1 (<a href="../../../LDC2012T06">LDC2012T06</a>)</li> <li>GALE Phase 2 Arabic Broadcast Conversation Parallel Text Part 2 (<a href="../../../LDC2012T14">LDC2012T14</a>)</li> <li>GALE Phase 2 Arabic Newswire Parallel Text (<a href="../../../LDC2012T17">LDC2012T17</a>)</li> <li>GALE Phase 2 Arabic Broadcast News Parallel Text (<a href="../../../LDC2012T18">LDC2012T18</a>)</li> <li>GALE Phase 2 Arabic Web Parallel Text (<a href="../../../LDC2013T01">LDC2013T01</a>)</li> </ul> <h3>Source Data</h3> A total of 10.7 hours of Arabic broadcast news recordings were selected from four sources and four different programs. A manual selection procedure was used to choose data appropriate for the GALE program, namely, news and conversation programs focusing on current events. Stories on topics such as sports, entertainment news, and stock market reports were excluded from the data set. The following table is a summary of the files included in this release. <table> <tbody> <tr> <td width="147">Source</td> <td width="125">Program</td> <td width="162">Epoch (YYYY.MM)</td> <td width="96">#hours</td> <td width="105">#words</td> </tr> <tr> <td>Dubai TV</td> <td>Dubai News</td> <td>2005.02 - 2005.12</td> <td>2.0</td> <td>11,078</td> </tr> <tr> <td>Nile TV</td> <td>News</td> <td>2000.11 - 2000.12</td> <td>0.5</td> <td>3,079</td> </tr> <tr> <td>Radio Sawa</td> <td>News at 06:00</td> <td>2005.11</td> <td>2.8</td> <td>9,712</td> </tr> <tr> <td>Voice of America</td> <td>News</td> <td>2000.11 - 2001.03</td> <td>5.4</td> <td>32,305</td> </tr> </tbody> </table> <h3>Transcription</h3> The selected audio files were carefully transcribed by LDC annotators and professional transcription agencies following LDC's Quick Rich Transcription specification. Manual sentence units/segments (SU) annotation was also performed as part of the transcription task. Three types of end of sentence SU are identified: - statement SU - question SU - incomplete SU <h3>Translation</h3> After transcription and SU annotation, files were reformatted into a human-readable translation format and were assigned to professional translators for careful translation. Translators followed LDC's GALE Translation guidelines, which describe the makeup of the translation team, the source data format, the translation data format, best practices for translating certain linguistic features (such as names and speech disfluencies), and quality control procedures applied to completed translations. <h3>Final Data</h3> TDF Format All final data are in Tab Delimited Format (TDF). TDF is compatible with other transcription formats, such as Transcriber format and AG format, and it is easy to process. Each line of a TDF file corresponds to a speech segment and contains 13 tab delimited fields (the 13th field "suType" might be empty): field data_type ----- --------- 1 file unicode 2 channel int 3 start float 4 end float 5 speaker unicode 6 speakerType unicode 7 speakerDialect unicode 8 transcript unicode 9 section int 10 turn int 11 segment int 12 sectionType unicode 13 suType unicode A source TDF file and its translation are the same except that the transcript in the source TDF is replaced by its English translation. Encoding All data are encoded in UTF8. <h3>Sponsorship</h3> This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. <h3>Samples</h3> For an example of the data in this corpus, please examine these screen captures of the <a href="desc/addenda/LDC2008T09_s.gif" rel="nofollow">source</a> and <a href="desc/addenda/LDC2008T09_t.gif" rel="nofollow">translation</a>. Portions © 2005 Dubai TV, © 2000 Nile TV, © 2005-2007, 2008 Trustees of the University of Pennsylvania

<h3>引言</h3> GALE第一阶段阿拉伯语广播新闻平行语料库（第二部分）是三卷本GALE第一阶段阿拉伯语广播新闻平行语料库的第二分册，该语料库与其他语料一同作为美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）资助的GALE项目第一年（第一阶段）的训练数据。<a href="http://catalog.ldc.upenn.edu/LDC2007T24" rel="nofollow">GALE第一阶段阿拉伯语广播新闻平行语料库（第一部分）</a>于2007年发布。 GALE第一阶段阿拉伯语广播新闻平行语料库（第二部分）包含从多渠道采集的10.7小时阿拉伯语广播新闻节目的转写文本与英文译文。本语料库不包含用于生成转写文本及译文的原始音频文件。 语言数据联盟（Linguistic Data Consortium, LDC）已发布以下GALE第一阶段与第二阶段阿拉伯语平行语料数据集： <ul> <li>GALE第一阶段阿拉伯语广播新闻平行语料库（第一部分）(<a href="../../../LDC2007T24">LDC2007T24</a>)</li> <li>GALE第一阶段阿拉伯语广播新闻平行语料库（第二部分）(<a href="../../../LDC2008T09">LDC2008T09</a>)</li> <li>GALE第一阶段阿拉伯语博客平行语料库(<a href="../../../LDC2008T02">LDC2008T02</a>)</li> <li>GALE第一阶段阿拉伯语新闻组平行语料库（第一部分）(<a href="../../../LDC2009T03">LDC2009T03</a>)</li> <li>GALE第一阶段阿拉伯语新闻组平行语料库（第二部分）(<a href="../../../LDC2009T09">LDC2009T09</a>)</li> <li>GALE第二阶段阿拉伯语广播访谈平行语料库（第一部分）(<a href="../../../LDC2012T06">LDC2012T06</a>)</li> <li>GALE第二阶段阿拉伯语广播访谈平行语料库（第二部分）(<a href="../../../LDC2012T14">LDC2012T14</a>)</li> <li>GALE第二阶段阿拉伯语新闻专线平行语料库(<a href="../../../LDC2012T17">LDC2012T17</a>)</li> <li>GALE第二阶段阿拉伯语广播新闻平行语料库(<a href="../../../LDC2012T18">LDC2012T18</a>)</li> <li>GALE第二阶段阿拉伯语网络平行语料库(<a href="../../../LDC2013T01">LDC2013T01</a>)</li> </ul> <h3>源数据</h3> 本次采集的10.7小时阿拉伯语广播新闻录音选自4个来源与4档不同节目。 本次数据采用人工遴选流程，选取适配GALE项目需求的内容，即聚焦时事的新闻与访谈节目。涉及体育、娱乐新闻、股市报道等主题的内容均被排除在本数据集之外。下表为本批次发布文件的汇总信息。 <table> <tbody> <tr> <td width="147">来源</td> <td width="125">节目名称</td> <td width="162">时间区间（YYYY.MM）</td> <td width="96">时长/小时</td> <td width="105">词汇量</td> </tr> <tr> <td>迪拜电视台（Dubai TV）</td> <td>迪拜新闻（Dubai News）</td> <td>2005.02 - 2005.12</td> <td>2.0</td> <td>11,078</td> </tr> <tr> <td>尼罗河电视台（Nile TV）</td> <td>新闻（News）</td> <td>2000.11 - 2000.12</td> <td>0.5</td> <td>3,079</td> </tr> <tr> <td>萨瓦广播电台（Radio Sawa）</td> <td>早间6点新闻（News at 06:00）</td> <td>2005.11</td> <td>2.8</td> <td>9,712</td> </tr> <tr> <td>美国之音（Voice of America）</td> <td>新闻（News）</td> <td>2000.11 - 2001.03</td> <td>5.4</td> <td>32,305</td> </tr> </tbody> </table> <h3>转写处理</h3> 经遴选的音频文件由LDC标注人员与专业转写机构按照LDC的快速富转写（Quick Rich Transcription）规范进行严谨转写。转写任务同时包含人工语句单元（Speech Unit, SU）标注环节。本次共识别三类语句单元结束类型： - 陈述型语句单元 - 疑问型语句单元 - 不完整语句单元 <h3>译制流程</h3> 完成转写与语句单元标注后，文件将被重格式化为便于人工阅读的译制格式，并交由专业译员进行精准翻译。译员需遵循LDC发布的GALE译制指南，该指南涵盖译制团队构成、源数据格式、译制数据格式、特定语言特征（如专有名词与言语不流畅现象）的译制规范，以及针对完成译稿的质量管控流程。 <h3>最终数据</h3> TDF格式 所有最终数据均采用制表符分隔格式（Tab Delimited Format, TDF）。该格式兼容Transcriber格式、AG格式等其他转写格式，且易于处理。 TDF文件的每一行对应一段语音片段，包含13个制表符分隔的字段（第13个字段“suType”可为空）： 字段序号字段名称数据类型 1 文件统一码（Unicode） 2 声道整数（int） 3 起始时间浮点数（float） 4 结束时间浮点数（float） 5 说话人统一码（Unicode） 6 说话人类型统一码（Unicode） 7 说话人口音统一码（Unicode） 8 转写文本统一码（Unicode） 9 章节整数（int） 10 话轮整数（int） 11 片段整数（int） 12 章节类型统一码（Unicode） 13 语句单元类型（suType）统一码（Unicode） 源语言TDF文件与对应的译制文件结构一致，仅将源语言TDF中的转写文本替换为英文译文。 编码方式 所有数据均采用UTF8编码格式。 <h3>资助说明</h3> 本项目部分由美国国防高级研究计划局（Defense Advanced Research Projects Agency）通过GALE项目资助（资助编号HR0011-06-1-0003）。本出版物内容未必代表美国政府的立场或政策，不应被视为获得官方背书。 <h3>数据样例</h3> 若需查看本语料库的数据示例，请参阅<a href="desc/addenda/LDC2008T09_s.gif" rel="nofollow">源文件</a>与<a href="desc/addenda/LDC2008T09_t.gif" rel="nofollow">译制文件</a>的屏幕截图。 部分内容 © 2005 迪拜电视台、© 2000 尼罗河电视台、© 2005-2007、2008 宾夕法尼亚大学托管会

提供机构：

Linguistic Data Consortium

创建时间：

2020-11-30

搜集汇总

背景与挑战

背景概述

该数据集是GALE计划第一阶段阿拉伯语广播新闻平行文本的第二部分，包含10.7小时来自多个来源（如迪拜电视台、美国之音）的阿拉伯语新闻节目的人工转录和英语翻译，数据采用TDF格式并专注于当前事件，适用于机器翻译和语言处理研究。

以上内容由遇见数据集搜集并总结生成