PTNews Corpus

The PTNews Corpus is a collection of over 19 million tokens extracted from 10 years of political news articles (in Portuguese) from the Portuguese newspaper PÚBLICO. The corpus is available under the Creative Commons Attribution-NonCommercial-ShareAlike Licence. The material contained on the PTNews Corpus is © 2010-2020 PÚBLICO Comunicação Social SA. The corpus sizes between the preprocessed version of Penn Treebank (PTB) and WikiText-103. Similarly to WikiText, PTNews has a larger vocabulary than PTB and retains the original case, punctuation and numbers. This corpus contains over 31000 publicly available full articles which makes it well suited for models that can take advantage of long-term dependencies. The corpus is available as a word-level collection of articles in two version: the first (ptnews_origin) contains a single file with all the articles in the form: title, URL, date, body; the second, contains only the title and body of the news articles and it is split into train, test, validation sets. In this processed version, the words with less than 3 occurrences are mapped to the <unk> token. Each sentence in an article body occupies a single line of the dataset and the end of paragraph is marked with the <eop> tag at the end of a sentence. Portuguese words resulting from contractions like "desta", ou "nesta" are separated into "d", "esta", "n", "esta", respectively. Sample article: Carlos César : Cavaco " cansado e sem entusiasmo " quis afastar responsabilidades sobre a crise https://publico.pt/2010/06/10/politica/noticia/carlos-cesar-cavaco-cansado-e-sem-entusiasmo-quis-afastar-responsabilidades-sobre-a-crise-1441369 2010-06-10 15:38:00 O presidente do Governo Regional dos Açores , Carlos César , considerou hoje que Cavaco Silva esteve " cansado e sem entusiasmo " no discurso do Dia de Portugal , onde afastou responsabilidades sobre a actual crise . <eop> " O país ouviu um Presidente cansado e sem entusiasmo , que andou às voltas com os papéis para dizer que não tinha nada a ver com as razões da crise " , afirmou Carlos César , num comentário à Lusa sobre o discurso do Presidente da República na cerimónia oficial do 10 de Junho , realizada em Faro . <eop> Carlos César considerou , no entanto , " positivo " que Cavaco Silva tenha feito " um discurso alinhado com um tema recorrente na apreciação do momento que vivemos , o da coesão e da corresponsabilização " . <eop> No mesmo sentido , manifestou concordância com o apelo que Cavaco Silva fez " à responsabilidade dos empregadores e empregados " , mas deixou um alerta relativamente à referência do Presidente da República à necessidade de " limpar Portugal " . <eop> Para Carlos César , se essa referência " for despida de conteúdo institucional útil , tratou-se de mais um discurso que se perderá na babugem política d aquilo que Cavaco Silva entendeu recordar como o ' rectângulo ' " . <eop> Reporting Results If you wish to report results or other resources obtained on the PTNews contact Davide Nunes with the following information: Task: e.g. Language Modelling, Semantic Similarity, etc; Publication URL: url to published article or preprint; Type of Model: LSTM Neural Network, n-grams, GloVe vectors, etc; Evaluation Metrics: e.g. validation and testing perplexities in the case of language modelling. They will be displayed here Preprocessed Corpus Statistics articles: 31.919 articles by split: train: 25.537 test: 3.191 val: 3.191 unique tokens: 68.318 unique OoV Tokens: 76.157 total tokens: 19.021.661 total OoV tokens: 95.043 OoV rate: 0.5% tokens by split: train: 15.242.995 test: 1.895.184 val: 1.883.482 Contact Information If you have questions about the corpus or want to report benchmark results, contact Davide Nunes.PTNews语料库（PTNews Corpus）是一个包含超1900万词元（Token）的语料集合，其数据源自葡萄牙报纸《PÚBLICO》十年间的政治新闻文章（葡萄牙语）。该语料库采用知识共享署名-非商业性使用-相同方式共享许可协议（Creative Commons Attribution-NonCommercial-ShareAlike Licence）进行发布，语料库内容版权归2010-2020年的PÚBLICO Comunicação Social SA所有。 该语料库的体量介于预处理版宾大树库（Penn Treebank, PTB）与WikiText-103之间。与WikiText类似，PTNews的词汇量大于PTB，且保留了原始的大小写格式、标点符号与数字信息。本语料库包含超过31000篇可公开获取的完整新闻文章，非常适合能够利用长距离依赖关系的模型开展训练。 该语料库以词级别的文章集合形式提供两种版本：第一种（ptnews_origin）为单文件整合形式，包含所有文章，格式为标题、URL、发布日期、正文；第二种仅包含新闻的标题与正文，并被划分为训练集、测试集与验证集。在该预处理版本中，出现次数少于3次的词会被映射为<unk>词元（Token）。新闻正文的每一个句子均单独占数据集的一行，段落结束则以<eop>标记置于句末。对于葡萄牙语中诸如“desta”“nesta”之类的缩合词汇，会被分别拆分为“d”“esta”与“n”“esta”，以此类推。 示例文章： 卡洛斯·塞萨尔：卡瓦科"疲惫且缺乏热情"欲规避危机责任 https://publico.pt/2010/06/10/politica/noticia/carlos-cesar-cavaco-cansado-e-sem-entusiasmo-quis-afastar-responsabilidades-sobre-a-crise-1441369 2010-06-10 15:38:00 亚速尔群岛政府区域主席卡洛斯·塞萨尔当日指出，卡瓦科·席尔瓦在葡萄牙日的演讲中"疲惫且缺乏热情"，并借此规避了其对当前危机应承担的责任。<eop> "全国民众听到了一位疲惫且缺乏热情的总统，他辗转腾挪只为表明自己与危机成因毫无关联"，卡洛斯·塞萨尔在向卢萨社（Lusa）发表关于共和国总统于6月10日在法鲁举行的官方纪念仪式上的演讲的评论时如此表示。<eop> 不过，卡洛斯·塞萨尔认为卡瓦科·席尔瓦的演讲"具有积极意义"，其围绕当下我们所处时代的反复出现的主题——凝聚力与共同责任——展开。<eop> 同理，他对卡瓦科·席尔瓦"呼吁雇主与雇员承担责任"的号召表示赞同，但对于共和国总统提及"整顿葡萄牙"的必要性，他发出了警示。<eop> 在卡洛斯·塞萨尔看来，如果该提及未能承载有益的制度内涵，那么这不过是又一篇消散在政治空谈中的演讲，也就是卡瓦科·席尔瓦口中所谓的"矩形（rectângulo）"。<eop> ### 成果上报 若您希望上报基于PTNews语料库得到的实验结果或其他相关资源，请联系戴维德·努内斯（Davide Nunes），并提供以下信息： - 任务类型：例如语言建模、语义相似度计算等； - 发表链接：已发表论文或预印本的URL； - 模型类型：例如长短期记忆网络（Long Short-Term Memory, LSTM）、n元语法模型、GloVe词向量等； - 评估指标：例如语言建模任务中的训练集与测试集困惑度。 上报的成果将在此处展示。 ### 预处理语料库统计信息 - 文章总数：31919篇 - 按划分的文章数： - 训练集：25537篇 - 测试集：3191篇 - 验证集：3191篇 - 唯一词元数：68318 - 未知词元（Out-of-Vocabulary, OoV）数：76157 - 总词元数：19021661 - 总未知词元数：95043 - 未知词元占比：0.5% - 按划分的词元数： - 训练集：15242995 - 测试集：1895184 - 验证集：1883482 ### 联系方式 若您对该语料库存在疑问，或希望上报基准测试结果，请联系戴维德·努内斯。