codeparrot/apps
收藏资源简介：
--- annotations_creators: [] language_creators: - crowdsourced - expert-generated language: ["code"] license: - mit multilinguality: - monolingual pretty_name: APPS size_categories: - unknown source_datasets: [] task_categories: - text-generation task_ids: - language-modeling --- # APPS Dataset ## Dataset Description [APPS](https://arxiv.org/abs/2105.09938) is a benchmark for code generation with 10000 problems. It can be used to evaluate the ability of language models to generate code from natural language specifications. You can also find **APPS metric** in the hub here [codeparrot/apps_metric](https://huggingface.co/spaces/codeparrot/apps_metric). ## Languages The dataset contains questions in English and code solutions in Python. ## Dataset Structure ```python from datasets import load_dataset load_dataset("codeparrot/apps") DatasetDict({ train: Dataset({ features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'], num_rows: 5000 }) test: Dataset({ features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'], num_rows: 5000 }) }) ``` ### How to use it You can load and iterate through the dataset with the following two lines of code for the train split: ```python from datasets import load_dataset import json ds = load_dataset("codeparrot/apps", split="train") sample = next(iter(ds)) # non-empty solutions and input_output features can be parsed from text format this way: sample["solutions"] = json.loads(sample["solutions"]) sample["input_output"] = json.loads(sample["input_output"]) print(sample) #OUTPUT: { 'problem_id': 0, 'question': 'Polycarp has $n$ different binary words. A word called binary if it contains only characters \'0\' and \'1\'. For example...', 'solutions': ["for _ in range(int(input())):\n n = int(input())\n mass = []\n zo = 0\n oz = 0\n zz = 0\n oo = 0\n...",...], 'input_output': {'inputs': ['4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n'], 'outputs': ['1\n3 \n-1\n0\n\n2\n1 2 \n']}, 'difficulty': 'interview', 'url': 'https://codeforces.com/problemset/problem/1259/D', 'starter_code': ''} } ``` Each sample consists of a programming problem formulation in English, some ground truth Python solutions, test cases that are defined by their inputs and outputs and function name if provided, as well as some metadata regarding the difficulty level of the problem and its source. If a sample has non empty `input_output` feature, you can read it as a dictionary with keys `inputs` and `outputs` and `fn_name` if it exists, and similarily you can parse the solutions into a list of solutions as shown in the code above. You can also filter the dataset for the difficulty level: Introductory, Interview and Competition. Just pass the list of difficulties as a list. E.g. if you want the most challenging problems, you need to select the competition level: ```python ds = load_dataset("codeparrot/apps", split="train", difficulties=["competition"]) print(next(iter(ds))["question"]) #OUTPUT: """\ Codefortia is a small island country located somewhere in the West Pacific. It consists of $n$ settlements connected by ... For each settlement $p = 1, 2, \dots, n$, can you tell what is the minimum time required to travel between the king's residence and the parliament house (located in settlement $p$) after some roads are abandoned? -----Input----- The first line of the input contains four integers $n$, $m$, $a$ and $b$ ... -----Output----- Output a single line containing $n$ integers ... -----Examples----- Input 5 5 20 25 1 2 25 ... Output 0 25 60 40 20 ... ``` ### Data Fields |Field|Type|Description| |---|---|---| |problem_id|int|problem id| |question|string|problem description| |solutions|string|some python solutions| |input_output|string|Json string with "inputs" and "outputs" of the test cases, might also include "fn_name" the name of the function| |difficulty|string|difficulty level of the problem| |url|string|url of the source of the problem| |starter_code|string|starter code to include in prompts| we mention that only few samples have `fn_name` and `starter_code` specified ### Data Splits The dataset contains a train and test splits with 5000 samples each. ### Dataset Statistics * 10000 coding problems * 131777 test cases * all problems have a least one test case except 195 samples in the train split * for tests split, the average number of test cases is 21.2 * average length of a problem is 293.2 words * all files have ground-truth solutions except 1235 samples in the test split ## Dataset Creation To create the APPS dataset, the authors manually curated problems from open-access sites where programmers share problems with each other, including Codewars, AtCoder, Kattis, and Codeforces. For more details please refer to the original [paper](https://arxiv.org/pdf/2105.09938.pdf). ## Considerations for Using the Data In [AlphaCode](https://arxiv.org/pdf/2203.07814v1.pdf) the authors found that this dataset can generate many false positives during evaluation, where incorrect submissions are marked as correct due to lack of test coverage. ## Citation Information ``` @article{hendrycksapps2021, title={Measuring Coding Challenge Competence With APPS}, author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt}, journal={NeurIPS}, year={2021} } ```
annotations_creators: [] language_creators: - 众包 - 专家生成 language: ["代码"] license: - MIT multilinguality: - 单语言 pretty_name: APPS size_categories: - 未知 source_datasets: [] task_categories: - 文本生成 task_ids: - 语言建模 # APPS 数据集 ## 数据集描述 [APPS](https://arxiv.org/abs/2105.09938) 是一个包含10000道题目的代码生成基准测试集，可用于评估大语言模型（Large Language Model）从自然语言规范生成代码的能力。你也可以在Hub上找到**APPS评测指标**，地址为[codeparrot/apps_metric](https://huggingface.co/spaces/codeparrot/apps_metric)。 ## 语言覆盖 该数据集包含英文的问题描述与Python代码解决方案。 ## 数据集结构 python from datasets import load_dataset load_dataset("codeparrot/apps") DatasetDict({ train: Dataset({ features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'], num_rows: 5000 }) test: Dataset({ features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'], num_rows: 5000 }) }) ### 如何使用该数据集 你可以通过以下两行代码加载并遍历训练划分的数据集： python from datasets import load_dataset import json ds = load_dataset("codeparrot/apps", split="train") sample = next(iter(ds)) # 可通过如下方式解析非空的solutions与input_output字段： sample["solutions"] = json.loads(sample["solutions"]) sample["input_output"] = json.loads(sample["input_output"]) print(sample) # 输出： { "problem_id": 0, "question": "Polycarp has $n$ different binary words. A word called binary if it contains only characters '0' and '1'. For example...", "solutions": ["for _ in range(int(input())): n = int(input()) mass = [] zo = 0 oz = 0 zz = 0 oo = 0 ...", ...], "input_output": {"inputs": ["4 4 0001 1000 0011 0111 3 010 101 0 2 00000 00001 4 01 001 0001 00001 "], "outputs": ["1 3 -1 0 2 1 2 "]}, "difficulty": "interview", "url": "https://codeforces.com/problemset/problem/1259/D", "starter_code": ""} } 每个样本包含一份英文编程问题描述、若干Python基准解决方案、由输入输出对定义的测试用例（若提供则包含函数名），以及标注问题难度等级与来源的元数据。 若样本的`input_output`字段非空，你可将其解析为包含`inputs`、`outputs`（若存在则包含`fn_name`字段）的字典；同理，你也可如上述代码所示，将solutions字段解析为解决方案列表。 你还可以按难度等级过滤数据集：入门级（Introductory）、面试级（Interview）与竞赛级（Competition）。只需将难度列表作为参数传入即可。例如，若你需要难度最高的竞赛级题目： python ds = load_dataset("codeparrot/apps", split="train", difficulties=["competition"]) print(next(iter(ds))["question"]) # 输出： """ Codefortia is a small island country located somewhere in the West Pacific. It consists of $n$ settlements connected by ... For each settlement $p = 1, 2, dots, n$, can you tell what is the minimum time required to travel between the king's residence and the parliament house (located in settlement $p$) after some roads are abandoned? -----Input----- The first line of the input contains four integers $n$, $m$, $a$ and $b$ ... -----Output----- Output a single line containing $n$ integers ... -----Examples----- Input 5 5 20 25 1 2 25 ... Output 0 25 60 40 20 ... """ ### 数据字段 |字段|类型|描述| |---|---|---| |problem_id|int|问题编号| |question|string|问题描述文本| |solutions|string|若干Python代码解决方案| |input_output|string|包含测试用例输入、输出（若存在则包含函数名fn_name）的JSON字符串| |difficulty|string|问题难度等级| |url|string|问题来源链接| |starter_code|string|可嵌入提示词的起始代码| 需注意，仅少数样本包含`fn_name`与`starter_code`字段。 ### 数据划分 该数据集包含训练集与测试集两个划分，各含5000个样本。 ### 数据集统计信息 * 10000道编程题目 * 131777个测试用例 * 除训练集中的195个样本外，所有问题均至少包含一个测试用例 * 测试集的平均测试用例数量为21.2 * 单条问题的平均长度为293.2个单词 * 除测试集中的1235个样本外，所有文件均包含基准解决方案 ## 数据集构建 APPS数据集的构建者从程序员共享编程题的开源平台（包括Codewars、AtCoder、Kattis及Codeforces）中手动筛选整理题目。更多细节可参阅原[论文](https://arxiv.org/pdf/2105.09938.pdf)。 ## 数据使用注意事项 在[AlphaCode](https://arxiv.org/pdf/2203.07814v1.pdf)相关论文中，作者发现该数据集在评估时会产生较多假阳性结果，即由于测试用例覆盖不足，错误的代码提交会被判定为正确。 ## 引用信息 bibtex @article{hendrycksapps2021, title={Measuring Coding Challenge Competence With APPS}, author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt}, journal={NeurIPS}, year={2021} }
数据集概述
数据集名称
- APPS
数据集描述
- APPS 是一个用于代码生成的基准数据集，包含10000个编程问题。该数据集用于评估语言模型从自然语言规范生成代码的能力。
语言
- 数据集包含英文问题和Python代码解决方案。
数据集结构
- 数据集分为训练集和测试集，各包含5000个样本。
- 每个样本包含以下特征：
problem_id: 问题ID（整数）question: 问题描述（字符串）solutions: Python解决方案（字符串）input_output: 测试案例的输入输出（字符串，可能包含函数名）difficulty: 问题难度级别（字符串）url: 问题来源的URL（字符串）starter_code: 提示中包含的起始代码（字符串）
数据集统计
- 总问题数：10000
- 测试案例数：131777
- 训练集中有195个样本没有测试案例
- 测试集中平均每个样本的测试案例数：21.2
- 问题描述的平均长度：293.2字
- 测试集中有1235个样本没有解决方案
数据集创建
- 数据集由作者从多个开放访问的编程问题共享网站手动筛选和整理而成，包括Codewars, AtCoder, Kattis, 和 Codeforces。
使用注意事项
- 在评估中可能会产生许多误报，因为测试覆盖不足可能导致错误的提交被标记为正确。
引用信息
@article{hendrycksapps2021, title={Measuring Coding Challenge Competence With APPS}, author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt}, journal={NeurIPS}, year={2021} }
