google/code_x_glue_tt_text_to_text

Name: google/code_x_glue_tt_text_to_text
Creator: google
Published: 2024-01-24 15:18:44
License: 暂无描述

Hugging Face2024-01-24 更新2024-05-25 收录

下载链接：

https://hf-mirror.com/datasets/google/code_x_glue_tt_text_to_text

下载链接

链接失效反馈

官方服务：

资源简介：

--- annotations_creators: - found language_creators: - found language: - da - en - lv - nb - zh license: - c-uda multilinguality: - multilingual size_categories: - 10K<n<100K source_datasets: - original task_categories: - translation task_ids: [] pretty_name: CodeXGlueTtTextToText tags: - code-documentation-translation dataset_info: - config_name: da_en features: - name: id dtype: int32 - name: source dtype: string - name: target dtype: string splits: - name: train num_bytes: 8163175 num_examples: 42701 - name: validation num_bytes: 190332 num_examples: 1000 - name: test num_bytes: 190772 num_examples: 1000 download_size: 4322666 dataset_size: 8544279 - config_name: lv_en features: - name: id dtype: int32 - name: source dtype: string - name: target dtype: string splits: - name: train num_bytes: 3644111 num_examples: 18749 - name: validation num_bytes: 192511 num_examples: 1000 - name: test num_bytes: 190867 num_examples: 1000 download_size: 1997959 dataset_size: 4027489 - config_name: no_en features: - name: id dtype: int32 - name: source dtype: string - name: target dtype: string splits: - name: train num_bytes: 8761755 num_examples: 44322 - name: validation num_bytes: 203815 num_examples: 1000 - name: test num_bytes: 197127 num_examples: 1000 download_size: 4661188 dataset_size: 9162697 - config_name: zh_en features: - name: id dtype: int32 - name: source dtype: string - name: target dtype: string splits: - name: train num_bytes: 9592148 num_examples: 50154 - name: validation num_bytes: 192147 num_examples: 1000 - name: test num_bytes: 195237 num_examples: 1000 download_size: 4733144 dataset_size: 9979532 configs: - config_name: da_en data_files: - split: train path: da_en/train-* - split: validation path: da_en/validation-* - split: test path: da_en/test-* - config_name: lv_en data_files: - split: train path: lv_en/train-* - split: validation path: lv_en/validation-* - split: test path: lv_en/test-* - config_name: no_en data_files: - split: train path: no_en/train-* - split: validation path: no_en/validation-* - split: test path: no_en/test-* - config_name: zh_en data_files: - split: train path: zh_en/train-* - split: validation path: zh_en/validation-* - split: test path: zh_en/test-* --- # Dataset Card for "code_x_glue_tt_text_to_text" ## Table of Contents - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits-sample-size) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** https://github.com/microsoft/CodeXGLUE/tree/main/Text-Text/text-to-text - **Paper:** https://arxiv.org/abs/2102.04664 ### Dataset Summary CodeXGLUE text-to-text dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Text-Text/text-to-text The dataset we use is crawled and filtered from Microsoft Documentation, whose document located at https://github.com/MicrosoftDocs/. ### Supported Tasks and Leaderboards - `machine-translation`: The dataset can be used to train a model for translating Technical documentation between languages. ### Languages da_en, lv_en, no_en, zh_en ## Dataset Structure ### Data Instances #### da_en An example of 'test' looks as follows. ``` { "id": 0, "source": "4 . K\u00f8r modellen , og udgiv den som en webtjeneste .\n", "target": "4 . Run the model , and publish it as a web service .\n" } ``` #### lv_en An example of 'train' looks as follows. ``` { "id": 0, "source": "title : Pakalpojumu objektu izveide\n", "target": "title : Create service objects\n" } ``` #### no_en An example of 'validation' looks as follows. ``` { "id": 0, "source": "2 . \u00c5pne servicevaren du vil definere komponenter fra en stykkliste for .\n", "target": "2 . Open the service item for which you want to set up components from a BOM .\n" } ``` #### zh_en An example of 'validation' looks as follows. ``` { "id": 0, "source": "& # 124 ; MCDUserNotificationReadStateFilterAny & # 124 ; 0 & # 124 ; \u5305\u62ec \u901a\u77e5 , \u800c \u4e0d \u8003\u8651 \u8bfb\u53d6 \u72b6\u6001 \u3002 & # 124 ;\n", "target": "| MCDUserNotificationReadStateFilterAny | 0 | Include notifications regardless of read state . |\n" } ``` ### Data Fields In the following each data field in go is explained for each config. The data fields are the same among all splits. #### da_en, lv_en, no_en, zh_en |field name| type | description | |----------|------|----------------------------------------| |id |int32 | The index of the sample | |source |string| The source language version of the text| |target |string| The target language version of the text| ### Data Splits |name |train|validation|test| |-----|----:|---------:|---:| |da_en|42701| 1000|1000| |lv_en|18749| 1000|1000| |no_en|44322| 1000|1000| |zh_en|50154| 1000|1000| ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators https://github.com/microsoft, https://github.com/madlag ### Licensing Information Computational Use of Data Agreement (C-UDA) License. ### Citation Information ``` @article{DBLP:journals/corr/abs-2102-04664, author = {Shuai Lu and Daya Guo and Shuo Ren and Junjie Huang and Alexey Svyatkovskiy and Ambrosio Blanco and Colin B. Clement and Dawn Drain and Daxin Jiang and Duyu Tang and Ge Li and Lidong Zhou and Linjun Shou and Long Zhou and Michele Tufano and Ming Gong and Ming Zhou and Nan Duan and Neel Sundaresan and Shao Kun Deng and Shengyu Fu and Shujie Liu}, title = {CodeXGLUE: {A} Machine Learning Benchmark Dataset for Code Understanding and Generation}, journal = {CoRR}, volume = {abs/2102.04664}, year = {2021} } ``` ### Contributions Thanks to @madlag (and partly also @ncoop57) for adding this dataset.

提供机构：

google

原始信息汇总

数据集概述

名称: CodeXGlueTtTextToText
语言: 多语言（da, en, lv, nb, zh）
许可证: c-uda
多语言性: 多语言
大小类别: 10K<n<100K
源数据集: 原始
任务类别: 翻译
标签: code-documentation-translation

数据集结构

数据实例

字段:
- id: int32
- source: string
- target: string

数据分割

名称	训练	验证	测试
da_en	42701	1000	1000
lv_en	18749	1000	1000
no_en	44322	1000	1000
zh_en	50154	1000	1000

数据大小

配置名称	下载大小	数据集大小
da_en	4322666	8544279
lv_en	1997959	4027489
no_en	4661188	9162697
zh_en	4733144	9979532

数据集创建

许可证信息: 计算数据使用协议（C-UDA）许可证。
引用信息:

@article{DBLP:journals/corr/abs-2102-04664, author = {Shuai Lu and Daya Guo and Shuo Ren and Junjie Huang and Alexey Svyatkovskiy and Ambrosio Blanco and Colin B. Clement and Dawn Drain and Daxin Jiang and Duyu Tang and Ge Li and Lidong Zhou and Linjun Shou and Long Zhou and Michele Tufano and Ming Gong and Ming Zhou and Nan Duan and Neel Sundaresan and Shao Kun Deng and Shengyu Fu and Shujie Liu}, title = {CodeXGLUE: {A} Machine Learning Benchmark Dataset for Code Understanding and Generation}, journal = {CoRR}, volume = {abs/2102.04664}, year = {2021} }

搜集汇总

数据集介绍

构建方式

在机器翻译与代码文档理解交叉领域中，高质量平行语料库的构建至关重要。google/code_x_glue_tt_text_to_text数据集源自微软官方文档库（MicrosoftDocs），经由系统化的网络爬取与过滤流程精心打造而成。该数据集涵盖了四种语言对，即丹麦语-英语、拉脱维亚语-英语、挪威语-英语以及中文-英语，每个样本均包含唯一的标识符、源语言文本与目标语言文本，从而构成了结构清晰的双语对齐资源。

特点

该数据集呈现出鲜明的多语言与专业化特性。其规模介于10,000至100,000条样本之间，各语言对的训练集容量从18,749至50,154条不等，验证集与测试集则均固定为1,000条，确保了评估的稳定与公平。尤为值得一提的是，数据内容聚焦于微软技术文档，涵盖操作指南、界面描述等专业领域文本，这使其在技术文档翻译任务中具有不可替代的领域适配性。

使用方法

研究人员可通过HuggingFace Datasets库便捷地调用该资源。加载时需指定具体配置名称，如'da_en'、'lv_en'、'no_en'或'zh_en'，以获取对应语言对的平行语料。数据集已预先划分为训练、验证与测试三个标准子集，可直接用于序列到序列模型的训练与评估。在机器翻译、跨语言代码文档理解等任务中，该数据集可作为基准测试的黄金标准，助力模型在技术文档翻译领域的性能提升。

背景与挑战

背景概述

在自然语言处理与软件工程交叉领域，代码与文档的跨语言翻译对于全球化技术生态的构建至关重要。由微软研究院于2021年发布的CodeXGLUE基准数据集，旨在系统性地评估代码理解与生成任务，其中文本到文本子集（code_x_glue_tt_text_to_text）专注于技术文档的多语言翻译。该数据集由微软团队（包括Shuai Lu、Daya Guo等研究者）从微软官方文档库中爬取并精心过滤，覆盖丹麦语、拉脱维亚语、挪威语及中文到英语的翻译对，总计超过15万条样本。其核心研究问题在于弥合低资源语言在技术文档领域的翻译鸿沟，为机器翻译模型提供领域适配的训练与评测基准，推动了跨语言技术知识传播的标准化进程。

当前挑战

该数据集所解决的领域挑战主要集中于技术文档翻译的专业性与多语言不平衡性。技术文本包含大量领域术语、代码片段及结构化指令，通用翻译模型常因缺乏领域语料而出现语义失真，且低资源语言（如拉脱维亚语）的标注数据稀缺，加剧了模型泛化难度。构建过程中的挑战则体现在数据采集与清洗层面：原始文档来源于异构的微软技术库，需统一格式并去除噪声（如HTML标签、非标准编码），同时确保翻译对在语义和语境上的严格对齐。此外，数据集仅提供约4万至5万条训练样本，对于深度神经网络而言规模有限，可能导致模型在长尾术语和复杂句式上的过拟合风险，制约了零样本翻译能力的突破。

常用场景

经典使用场景

在自然语言处理与软件工程的交叉领域，技术文档的跨语言翻译长期面临术语专业性强、句式结构复杂等挑战。CodeXGLUE文本到文本数据集应运而生，其经典应用场景聚焦于构建多语言技术文档的神经机器翻译模型。该数据集提供了丹麦语、拉脱维亚语、挪威语及中文至英文的高质量平行语料，涵盖微软官方技术文档中的实例化句子，为训练端到端的序列到序列模型提供了标准化基准。研究者可基于此数据集探索低资源语言翻译、领域自适应翻译及代码相关文本的语义对齐等任务，推动技术文档全球化进程中的语言障碍消解。

衍生相关工作

基于该数据集，学术界涌现了一系列具有代表性的衍生工作。其中，微软研究院提出的CodeXGLUE基准套件将文本到文本翻译任务与代码补全、缺陷检测等任务并列，构建了统一的评估框架，推动了多任务学习在软件工程领域的应用。后续研究如采用预训练语言模型（如mBART、XLM-R）进行微调，验证了跨语言文档翻译中零样本迁移的可行性。另有工作聚焦于技术文档的术语对齐与回译增强策略，通过数据增强手段提升低资源语言对的翻译质量。这些衍生研究不仅深化了对技术文档语言特性的理解，也为构建更鲁棒的代码智能系统奠定了方法论基础。

数据集最近研究