Nos_CorpusNOS-GL: Galician Macrocorpus for LLM training

Update notice

(EN)

An updated version of CorpusNÓS is available on Hugging Face at: https://huggingface.co/datasets/proxectonos/corpusnos

Future updates, corrections, and extended releases of the corpus will be primarily made available through the Hugging Face repository. This Zenodo record is kept for archival and citation purposes, but users are encouraged to consult the Hugging Face version for the most recent release.

(GL)

Unha versión actualizada de CorpusNÓS está dispoñible en Hugging Face: https://huggingface.co/datasets/proxectonos/corpusnos

As futuras actualizacións, correccións e versións ampliadas do corpus publicaranse principalmente a través do repositorio de Hugging Face. Este rexistro de Zenodo mantense con fins de arquivo e citación, mais recoméndase ás persoas usuarias consultar a versión de Hugging Face para acceder á publicación máis recente.

-----------------

CorpusNÓS is a massive Galician corpus made up of 2.1B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. 

------------------

We happily announce that we are introducing a new version of the CorpusNÓS. After improving our text cleaning and processing methods in our cleaning pipeline, we have decided to release this new version of the corpus, which reflects those enhancements. 

This new version contains the same files as the previous one and holds the same distribution of the data, however, we decided to change the format from plain text (*.txt) to JSONL (*.jsonl) so future cleaning processes can be performed easily, and relevant metadata can be included. As of now, some examples of entries from the CorpusNós have the following structure: 

{"id": 0, "text": "Abades: Parroquia do concello de Baltar baixo a advocación de san Paio.", "num_words": 12} 

{"id": 581, "text": "Feliz 2008 a tódolos nosos lectores\nAgora que remata 2007, un ano cheo de novidades tecnolóxicas que difundimos a través deste espazo dixital, queremos desexar a tódolos que non seguen con fidelidade unha boa despedida do ano e un feliz aninovo.\nNós volveremos o mércores, 2 de xaneiro, á nosa actividade ordinaria, cumprindo coa nosa labor informativa para que as novas tecnolóxicas de Galicia e en galego cheguen ós nosos lectores puntualmente.", "num_words": 72, "pyplexity_score": 717.7585757844212, "lang": "gl"} 

In the plain text version, the delimiter between different documents was constituted by two newlines (\n\n). In the JSONL version, each document is a JSON object with their corresponding id, but it also includes the number of words of each document, and, in some cases, the pyplexity score and the language tag. 

This new version of CorpusNós has undergone a heavier process of deduplication than the previous one. This means that more exact match duplications as well as partial duplications have been removed from the corpus and, therefore, the number of documents and tokens in this version has decreased and the current statistics are: 

Subcorpus: 

Data obtained via transfer agreement 

Genre 

Nº tokens 

Nº documents 

Books 

7.217.626 

103 

Research articles 

2.638.537 

635 

Press 

92.661.350 

161.760 

Governmental 

221.565.059 

527.699 

Web contents 

15.471.132 

41.276 

Encyclopedic 

4.799.214 

47.396 

Subtotal 

332.721.231 

777.583 

Subcorpus: 

Public data 

Genre 

Nº tokens 

Nº documents 

Press and blogs 

142.238.181 

598.375 

Encyclopedic 

48.260.708 

148.560 

Web crawls 

1.205.699.835 

2.850.604 

Translation corpora 

106.555.883 

3.544.026 

Subtotal 

1.502.754.607 

7.141.565 

Total 

1.835.475.838 

7.919.148 

The TXT version is still available under the corpusnos_v1_txt zip file and it mantains the same structure as before (documents are divided by two newlines '\n\n') but this version hasn't gone through the improved cleaning process mentioned above. 

Please, note that if you want to download or use the newest version you have to download the corpusnos_v2_jsonl.

Note: Some of the files referenced may be missing in this version of the corpus due to pending transfer agreements and they will be included in a future version of the corpus as soon as they are available for publishing.  Note: Please, note that the following subcorpora have different licenses which correspond to their original licenses as specified in the paper: TED2020 (CC BY–NC–ND 4.0), mC4 (Apache License 2.0), OSCAR (CC0).  

Please refer to our paper for more details, CorpusNÓS: A massive Galician corpus for training large language models. 

If you use this data in your work, please cite: 

de-Dios-Flores, Iria, Silvia Paniagua Suárez, Cristina Carbajal Pérez, Daniel Bardanca Outeiriño, Marcos Garcia and Pablo Gamallo. 2024. CorpusNÓS: A massive Galician corpus for training large language models. Proceedings of the 16th International Conference on Computational Processing of Portuguese - ACL Anthology (Volume 1), 593-599. 

Funding

This corpus was compiled/development.... within the Nós Project, funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the [project ILENIA] (https://proyectoilenia.es/) with reference 2022/TL22/00215336.