Helsinki-NLP/opus_gnome

--- annotations_creators: - found language_creators: - found language: - af - am - an - ang - ar - as - ast - az - bal - be - bem - bg - bn - bo - br - brx - bs - ca - crh - cs - csb - cy - da - de - dv - dz - el - en - eo - es - et - eu - fa - fi - fo - fr - fur - fy - ga - gd - gl - gn - gu - gv - ha - he - hi - hr - hu - hy - ia - id - ig - io - is - it - ja - jbo - ka - kg - kk - km - kn - ko - kr - ks - ku - ky - la - lg - li - lo - lt - lv - mai - mg - mi - mk - ml - mn - mr - ms - mt - mus - my - nb - nds - ne - nhn - nl - nn - 'no' - nqo - nr - nso - oc - or - os - pa - pl - ps - pt - quz - ro - ru - rw - si - sk - sl - so - sq - sr - st - sv - sw - szl - ta - te - tg - th - tk - tl - tr - ts - tt - tyj - ug - uk - ur - uz - vi - wa - xh - yi - yo - zh - zu license: - unknown multilinguality: - multilingual size_categories: - 10K<n<100K - 1K<n<10K - n<1K source_datasets: - original task_categories: - translation task_ids: [] pretty_name: OpusGnome config_names: - ar-bal - bg-csb - ca-en_GB - cs-eo - cs-tk - da-vi - de-ha - de-tt - el-sk - en_GB-my language_bcp47: - ar-TN - az-IR - bg-BG - bn-IN - da-DK - de-CH - en-AU - en-CA - en-GB - en-NZ - en-US - en-ZA - es-AR - es-CL - es-CO - es-CR - es-DO - es-EC - es-ES - es-GT - es-HN - es-MX - es-NI - es-PA - es-PE - es-PR - es-SV - es-UY - es-VE - fa-IR - hi-IN - it-IT - ms-MY - nb-NO - nn-NO - no-NB - pt-BR - pt-PT - sr-ME - tg-TJ - tl-PH - tr-TR - ur-PK - vi-VN - zh-CN - zh-HK - zh-TW dataset_info: - config_name: ar-bal features: - name: id dtype: string - name: translation dtype: translation: languages: - ar - bal splits: - name: train num_bytes: 5138 num_examples: 60 download_size: 3695 dataset_size: 5138 - config_name: bg-csb features: - name: id dtype: string - name: translation dtype: translation: languages: - bg - csb splits: - name: train num_bytes: 172533 num_examples: 1768 download_size: 48120 dataset_size: 172533 - config_name: ca-en_GB features: - name: id dtype: string - name: translation dtype: translation: languages: - ca - en_GB splits: - name: train num_bytes: 1007476 num_examples: 7982 download_size: 422733 dataset_size: 1007476 - config_name: cs-eo features: - name: id dtype: string - name: translation dtype: translation: languages: - cs - eo splits: - name: train num_bytes: 2883 num_examples: 73 download_size: 4183 dataset_size: 2883 - config_name: cs-tk features: - name: id dtype: string - name: translation dtype: translation: languages: - cs - tk splits: - name: train num_bytes: 1197707 num_examples: 18686 download_size: 387467 dataset_size: 1197707 - config_name: da-vi features: - name: id dtype: string - name: translation dtype: translation: languages: - da - vi splits: - name: train num_bytes: 9360 num_examples: 149 download_size: 8243 dataset_size: 9360 - config_name: de-ha features: - name: id dtype: string - name: translation dtype: translation: languages: - de - ha splits: - name: train num_bytes: 22887 num_examples: 216 download_size: 8846 dataset_size: 22887 - config_name: de-tt features: - name: id dtype: string - name: translation dtype: translation: languages: - de - tt splits: - name: train num_bytes: 134966 num_examples: 2169 download_size: 38009 dataset_size: 134966 - config_name: el-sk features: - name: id dtype: string - name: translation dtype: translation: languages: - el - sk splits: - name: train num_bytes: 12109 num_examples: 150 download_size: 9379 dataset_size: 12109 - config_name: en_GB-my features: - name: id dtype: string - name: translation dtype: translation: languages: - en_GB - my splits: - name: train num_bytes: 3298038 num_examples: 28232 download_size: 1045971 dataset_size: 3298038 configs: - config_name: ar-bal data_files: - split: train path: ar-bal/train-* - config_name: bg-csb data_files: - split: train path: bg-csb/train-* - config_name: ca-en_GB data_files: - split: train path: ca-en_GB/train-* - config_name: cs-eo data_files: - split: train path: cs-eo/train-* - config_name: cs-tk data_files: - split: train path: cs-tk/train-* - config_name: da-vi data_files: - split: train path: da-vi/train-* - config_name: de-ha data_files: - split: train path: de-ha/train-* - config_name: de-tt data_files: - split: train path: de-tt/train-* - config_name: el-sk data_files: - split: train path: el-sk/train-* - config_name: en_GB-my data_files: - split: train path: en_GB-my/train-* --- # Dataset Card for Opus Gnome ## Table of Contents - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** http://opus.nlpl.eu/GNOME.php - **Repository:** None - **Paper:** http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf - **Leaderboard:** [More Information Needed] - **Point of Contact:** [More Information Needed] ### Dataset Summary To load a language pair which isn't part of the config, all you need to do is specify the language code as pairs. You can find the valid pairs in Homepage section of Dataset Description: http://opus.nlpl.eu/GNOME.php E.g. `dataset = load_dataset("opus_gnome", lang1="it", lang2="pl")` ### Supported Tasks and Leaderboards [More Information Needed] ### Languages [More Information Needed] ## Dataset Structure ### Data Instances ``` { 'id': '0', 'translation': { 'ar': 'إعداد سياسة القفل', 'bal': 'تنظیم کتن سیاست کبل' } } ``` ### Data Fields Each instance has two fields: - **id**: the id of the example - **translation**: a dictionary containing translated texts in two languages. ### Data Splits Each subset simply consists in a train set. We provide the number of examples for certain language pairs: | | train | |:---------|--------:| | ar-bal | 60 | | bg-csb | 10 | | ca-en_GB | 7982 | | cs-eo | 73 | | de-ha | 216 | | cs-tk | 18686 | | da-vi | 149 | | en_GB-my | 28232 | | el-sk | 150 | | de-tt | 2169 | ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data [More Information Needed] #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations [More Information Needed] #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information [More Information Needed] ### Citation Information @InProceedings{TIEDEMANN12.463, author = {J{\"o}rg Tiedemann}, title = {Parallel Data, Tools and Interfaces in OPUS}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Ugur Dogan and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } ### Contributions Thanks to [@rkc007](https://github.com/rkc007) for adding this dataset.

原始信息汇总

数据集概述

基本信息

• 数据集名称: OpusGnome
• 语言: 多语言，包含多种语言对
• 许可: 未知
• 多语言性: 多语言
• 数据集大小: 1K<n<10K, 10K<n<100K, n<1K
• 源数据: 原始数据
• 任务类别: 翻译

数据集结构

数据实例

json { "id": "0", "translation": { "ar": "إعداد سياسة القفل", "bal": "تنظیم کتن سیاست کبل" } }

数据字段

• id: 示例的ID
• translation: 包含两种语言翻译文本的字典

数据分割

每个子集仅包含训练集。以下是部分语言对的训练集示例数量：

语言对	训练集数量
ar-bal	60
bg-csb	1768
ca-en_GB	7982
cs-eo	73
cs-tk	18686
da-vi	149
de-ha	216
de-tt	2169
el-sk	150
en_GB-my	28232

配置信息

以下是部分配置的详细信息：

ar-bal

• 特征:
  • id: string
  • translation:
    • languages: [ar, bal]
• 分割:
  • train:
    • num_bytes: 5138
    • num_examples: 60
• 下载大小: 3695
• 数据集大小: 5138

bg-csb

• 特征:
  • id: string
  • translation:
    • languages: [bg, csb]
• 分割:
  • train:
    • num_bytes: 172533
    • num_examples: 1768
• 下载大小: 48120
• 数据集大小: 172533

ca-en_GB

• 特征:
  • id: string
  • translation:
    • languages: [ca, en_GB]
• 分割:
  • train:
    • num_bytes: 1007476
    • num_examples: 7982
• 下载大小: 422733
• 数据集大小: 1007476

cs-eo

• 特征:
  • id: string
  • translation:
    • languages: [cs, eo]
• 分割:
  • train:
    • num_bytes: 2883
    • num_examples: 73
• 下载大小: 4183
• 数据集大小: 2883

cs-tk

• 特征:
  • id: string
  • translation:
    • languages: [cs, tk]
• 分割:
  • train:
    • num_bytes: 1197707
    • num_examples: 18686
• 下载大小: 387467
• 数据集大小: 1197707

da-vi

• 特征:
  • id: string
  • translation:
    • languages: [da, vi]
• 分割:
  • train:
    • num_bytes: 9360
    • num_examples: 149
• 下载大小: 8243
• 数据集大小: 9360

de-ha

• 特征:
  • id: string
  • translation:
    • languages: [de, ha]
• 分割:
  • train:
    • num_bytes: 22887
    • num_examples: 216
• 下载大小: 8846
• 数据集大小: 22887

de-tt

• 特征:
  • id: string
  • translation:
    • languages: [de, tt]
• 分割:
  • train:
    • num_bytes: 134966
    • num_examples: 2169
• 下载大小: 38009
• 数据集大小: 134966

el-sk

• 特征:
  • id: string
  • translation:
    • languages: [el, sk]
• 分割:
  • train:
    • num_bytes: 12109
    • num_examples: 150
• 下载大小: 9379
• 数据集大小: 12109

en_GB-my

• 特征:
  • id: string
  • translation:
    • languages: [en_GB, my]
• 分割:
  • train:
    • num_bytes: 3298038
    • num_examples: 28232
• 下载大小: 1045971
• 数据集大小: 3298038