Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered

---
dataset_info:
features:
- name: uuid
dtype: string
- name: model
dtype: string
- name: gen_input_configs
struct:
- name: temperature
dtype: float64
- name: top_p
dtype: float64
- name: input_generator
dtype: string
- name: seed
dtype: 'null'
- name: pre_query_template
dtype: string
- name: instruction
dtype: string
- name: response
dtype: string
- name: conversations
list:
- name: from
dtype: string
- name: value
dtype: string
- name: task_category
dtype: string
- name: other_task_category
sequence: string
- name: task_category_generator
dtype: string
- name: difficulty
dtype: string
- name: intent
dtype: string
- name: knowledge
dtype: string
- name: difficulty_generator
dtype: string
- name: input_quality
dtype: string
- name: quality_explanation
dtype: string
- name: quality_generator
dtype: string
- name: llama_guard_2
dtype: string
- name: reward_model
dtype: string
- name: instruct_reward
dtype: float64
- name: min_neighbor_distance
dtype: float64
- name: repeat_count
dtype: int64
- name: min_similar_uuid
dtype: string
- name: instruction_length
dtype: int64
- name: response_length
dtype: int64
- name: language
dtype: string
splits:
- name: train
num_bytes: 1656792825.9963841
num_examples: 300000
download_size: 1009928826
dataset_size: 1656792825.9963841
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
license: llama3.1
task_categories:
- text-generation
language:
- en
size_categories:
- 100K<n<1M
---


Project Web: [https://magpie-align.github.io/](https://magpie-align.github.io/)

Arxiv Technical Report: [https://arxiv.org/abs/2406.08464](https://arxiv.org/abs/2406.08464)

Codes: [https://github.com/magpie-align/magpie](https://github.com/magpie-align/magpie)

## Abstract
<details><summary>Click Here</summary>
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
</details><be>

## Dataset Details
This dataset is generated by [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) using [Magpie](https://huggingface.co/Magpie-Align). Please refer to our [paper](https://arxiv.org/abs/2406.08464) and [codebase](https://github.com/magpie-align/magpie) for implementation details.

**License**: Please follow [Meta Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE).

### Available Labels
- **Input Length**: The total number of characters in the instructions.
- **Output Length**: The total number of characters in the responses.
- **Task Category**: The specific category of the instructions.
- **Input Quality**: The clarity, specificity, and coherence of the instructions, rated as 'very poor', 'poor', 'average', 'good', and 'excellent'.
- **Input Difficulty**: The level of knowledge required to address the task described in the instruction, rated as 'very easy', 'easy', 'medium', 'hard', or 'very hard'.
- **Minimum Neighbor Distance**: The embedding distance to the nearest neighbor within the dataset. It can be used for filtering out repetitive or similar instances.
- **Safety**: Safety tags marked by [meta-llama/Meta-Llama-Guard-2-8B](https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B)
- **Reward**: The output of the reward model given the specific instruction-response pair.
- **Language**: The language of the instruction.

## Filter Setups
- **Input Quality**: >= good
- **Instruction Reward**: >=-10
- Remove repetition and incomplete instructions (e.g., end with :)
- Choose 300K data with the longest responses

## Limitations
This dataset contains a large amount of chain-of-thought responses, which may potentially decrease the performance. Therefore, we reduce the amount of data containing `## Step 1` in the multi-turn version: [Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-v0.1](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-v0.1).

## Dataset Navigation 🧭
|Model Name | Dataset | Type | Description |
|-------------|:-------|:-------|:-------|
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-1M](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-1M-v0.1) | SFT | 1M Raw conversations built with Meta Llama 3.1 70B.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered) | SFT | Apply a filter and select 300K high quality conversations.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-500K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-500K-Filtered) | SFT | Apply a filter and select 500K high quality conversations.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-MT-500K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-500K-v0.1) | SFT | Extend Magpie-Llama-3.1-Pro-500K-Filtered to multi-turn.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-MT-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-Filtered) | SFT | Select 300K high quality multi-turn conversations from Magpie-Llama-3.1-Pro-MT-500K.
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1) | DPO | DPO dataset via Best-of-N sampling and rewards.

数据集信息：
特征字段：
- 名称：uuid，数据类型：字符串
- 名称：model，数据类型：字符串
- 名称：gen_input_configs，数据类型：结构体，包含以下子字段：
- 名称：temperature（温度参数），数据类型：float64
- 名称：top_p（top-p采样参数），数据类型：float64
- 名称：input_generator（输入生成器），数据类型：字符串
- 名称：seed（随机种子），数据类型：null
- 名称：pre_query_template（预查询模板），数据类型：字符串
- 名称：instruction（指令），数据类型：字符串
- 名称：response（响应），数据类型：字符串
- 名称：conversations，数据类型：列表，列表内元素包含：
- 名称：from（对话来源），数据类型：字符串
- 名称：value（对话内容），数据类型：字符串
- 名称：task_category（任务类别），数据类型：字符串
- 名称：other_task_category（其他任务类别，序列类型），数据类型：字符串序列
- 名称：task_category_generator（任务类别生成器），数据类型：字符串
- 名称：difficulty（难度），数据类型：字符串
- 名称：intent（意图），数据类型：字符串
- 名称：knowledge（知识背景），数据类型：字符串
- 名称：difficulty_generator（难度生成器），数据类型：字符串
- 名称：input_quality（输入质量），数据类型：字符串
- 名称：quality_explanation（质量解释），数据类型：字符串
- 名称：quality_generator（质量生成器），数据类型：字符串
- 名称：llama_guard_2（Llama Guard 2），数据类型：字符串
- 名称：reward_model（奖励模型），数据类型：字符串
- 名称：instruct_reward（指令奖励得分），数据类型：float64
- 名称：min_neighbor_distance（最小近邻距离），数据类型：float64
- 名称：repeat_count（重复次数），数据类型：int64
- 名称：min_similar_uuid（最小相似uuid），数据类型：字符串
- 名称：instruction_length（指令长度），数据类型：int64
- 名称：response_length（响应长度），数据类型：int64
- 名称：language（语言），数据类型：字符串
数据拆分：
- 拆分名称：train（训练集），字节数：1656792825.9963841，样本数量：300000
下载大小：1009928826，数据集总大小：1656792825.9963841
配置信息：
- 配置名称：default，数据文件：
- 拆分：train（训练集），路径：data/train-*
许可证：Llama 3.1
任务类别：文本生成（text-generation）
语言：英语（en）
样本规模类别：100K < n < 1M


项目官网：[https://magpie-align.github.io/](https://magpie-align.github.io/)
Arxiv技术报告：[https://arxiv.org/abs/2406.08464](https://arxiv.org/abs/2406.08464)
代码仓库：[https://github.com/magpie-align/magpie](https://github.com/magpie-align/magpie)

## 摘要
<details><summary>点击展开</summary>
高质量指令数据对大语言模型（Large Language Model，LLM）的对齐至关重要。尽管部分模型（如Llama-3-Instruct）已开放权重，但其对齐数据仍处于私有状态，这阻碍了人工智能的民主化进程。当前的开源数据构建方法面临人工成本高昂、提示范围预先限定且有限的问题，难以实现有效扩展，进而可能限制了公开对齐数据集的多样性与质量。我们能否直接从已对齐的大语言模型中提取数据，以此大规模合成高质量的指令数据？为此，我们提出了一种用于生成大规模对齐数据的自合成方法，名为Magpie。我们的核心观察是：得益于自回归特性，当仅向Llama-3-Instruct这类已对齐大语言模型输入用户消息预留位置之前的左侧模板时，模型能够生成用户查询。我们利用该方法对Llama-3-Instruct进行提示，生成了400万条指令及其对应的响应。我们对提取的数据进行了全面分析，并筛选出30万个高质量样本。为了将Magpie数据集与其他公开指令数据集进行对比，我们分别使用每个数据集对Llama-3-8B-Base进行微调，并评估微调后模型的性能。结果表明，在部分任务中，使用Magpie数据集微调的模型性能可与官方的Llama-3-8B-Instruct相媲美——尽管后者通过监督微调（Supervised Fine-Tuning，SFT）与后续反馈学习，使用了1000万条数据进行增强。我们还证实，仅使用Magpie数据集进行监督微调，其性能便可超越此前用于监督微调和偏好优化的公开数据集，例如结合UltraFeedback的直接偏好优化（Direct Preference Optimization，DPO）。这一优势在AlpacaEval、ArenaHard与WildBench等对齐基准测试中均有体现。
</details>

## 数据集详情
本数据集由[Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)通过[Magpie](https://huggingface.co/Magpie-Align)生成。有关实现细节，请参阅我们的[论文](https://arxiv.org/abs/2406.08464)与[代码库](https://github.com/magpie-align/magpie)。
**许可证**：请遵循[Meta Llama 3.1社区许可证](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE)。

### 可用标签
- **输入长度**：指令中的总字符数。
- **输出长度**：响应中的总字符数。
- **任务类别**：指令所属的具体类别。
- **输入质量**：指令的清晰度、特异性与连贯性，评级分为「极差」「差」「一般」「良好」与「优秀」。
- **输入难度**：完成指令描述的任务所需的知识水平，评级分为「极简单」「简单」「中等」「困难」与「极困难」。
- **最小近邻距离**：数据集中与当前样本最近邻的嵌入距离，可用于过滤重复或相似样本。
- **安全性**：由[meta-llama/Meta-Llama-Guard-2-8B](https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B)标记的安全标签。
- **奖励得分**：针对特定指令-响应对的奖励模型输出结果。
- **语言**：指令所使用的语言。

## 筛选设置
- 输入质量：≥良好
- 指令奖励得分：≥-10
- 移除重复与不完整的指令（例如以「:」结尾的样本）
- 选取30万个响应最长的样本

## 局限性
本数据集包含大量思维链（Chain-of-Thought，CoT）响应，这可能会对模型性能产生负面影响。因此，我们在多轮版本[Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-v0.1](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-v0.1)中，减少了包含`## Step 1`的样本数量。

## 数据集导航 🧭
|模型名称 | 数据集 | 类型 | 描述 |
|-------------|:-------|:-------|:-------|
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-1M](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-1M-v0.1) | 监督微调（SFT） | 基于Meta Llama 3.1 70B构建的100万条原始对话数据。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered) | 监督微调（SFT） | 经过筛选后选取的30万个高质量对话数据。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-500K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-500K-Filtered) | 监督微调（SFT） | 经过筛选后选取的50万个高质量对话数据。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-MT-500K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-500K-v0.1) | 监督微调（SFT） | 将Magpie-Llama-3.1-Pro-500K-Filtered扩展为多轮对话格式。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-MT-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-Filtered) | 监督微调（SFT） | 从Magpie-Llama-3.1-Pro-MT-500K中选取的30万个高质量多轮对话数据。
| [Llama 3.1 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | [Magpie-Llama-3.1-Pro-DPO-100K](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1) | 直接偏好优化（DPO） | 通过最佳N采样与奖励机制构建的DPO数据集。