tollefj/massive-en-no-shorter-transfer

Massive EN-NO shorter transfer数据集是一个包含英语（EN）和挪威语（NO）翻译的数据集，数据来源于多个HuggingFace数据集，包括opus100、opus_books、open_subtitles和RuterNorway/Fleurs-Alpaca-EN-NO。数据集经过简单预处理、相似度计算和长度过滤等步骤，生成了一个更短且更相似的翻译语料库。预处理步骤包括去除/纠正错误的标点符号，使用NbAiLab/nb-sbert-base模型计算所有翻译对的相似度以对齐翻译，并过滤掉目标语言（挪威语）长度少于源语言（英语）长度70%的翻译对，除非翻译对中的单词数少于6个。

The Massive EN-NO Shorter Transfer dataset is a parallel translation corpus consisting of English (EN) and Norwegian (NO) sentence pairs. It is sourced from multiple Hugging Face datasets including opus100, opus_books, open_subtitles, and RuterNorway/Fleurs-Alpaca-EN-NO. To generate a shorter and semantically more aligned translation corpus, the dataset undergoes three core processing steps: simple preprocessing, similarity calculation, and length filtering. Specifically, the preprocessing step involves removing and correcting erroneous punctuation marks. Next, the NbAiLab/nb-sbert-base model is utilized to compute the semantic similarity of all translation pairs for alignment purposes. Finally, translation pairs where the length of the target language (Norwegian) is less than 70% of that of the source language (English) are filtered out, with an exception granted for pairs containing fewer than 6 words in total.

原始信息汇总

数据集概述

数据集信息

• 特征:
  • en: 类型为字符串
  • no: 类型为字符串
• 分割:
  • train: 字节数为44628652，样本数为758144
• 下载大小: 33446436字节
• 数据集大小: 44628652字节

配置

• 配置名称: default
  • 数据文件:
    • train: 路径为data/train-*

许可

• 许可: cc

任务类别

• 任务类别:
  • 翻译
  • 摘要

语言

• 语言:
  • no
  • nb
  • en

名称

• 名称: Massive EN-NO shorter transfer

大小类别

• 大小类别:
  • 100K<n<1M