essobi/trl_gair_lima_v19

---
dataset_info:
features:
- name: messages
list:
- name: role
dtype: string
- name: content
dtype: string
- name: metadata
dtype: json
splits:
- name: train
- name: test
download_size: unknown
dataset_size: unknown
license: cc-by-nc-4.0
---

# GAIR/LIMA TRL Dataset

This dataset is a transformed version of the [GAIR/LIMA](https://huggingface.co/datasets/GAIR/LIMA) dataset, specifically formatted for training with TRL (Transformers Reinforcement Learning).

## Dataset Description

- **Homepage:** [GAIR/LIMA](https://huggingface.co/datasets/GAIR/LIMA)
- **Repository:** [ProblemGenerationAgent](https://github.com/er-ads/ProblemGenerationAgent)
- **Paper:** [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206)

## Dataset Summary

This dataset contains conversations from the GAIR/LIMA dataset, transformed into a message-based format suitable for conversational AI training and reinforcement learning.

### Key Transformations

- Converted from Q&A format to multi-turn conversation format
- Applied message-level transformations (HTML tag removal, whitespace normalization)
- Filtered conversations based on quality criteria (length checks, content validation)
- Preserved original metadata for reference and traceability

## Dataset Structure

Each example contains:

- `messages`: List of conversation turns, each with:
- `role`: Either "user" or "assistant"
- `content`: Message text
- `metadata`: Original and derived metadata including:
- `original_id`: ID from original LIMA dataset
- `answers_count`: Number of answers in original format
- `original_data`: Preserved original fields

## Splits

- **train**: Training split (filtered LIMA train set)
- **test**: Test split (filtered LIMA test set)

## Creation Process

This dataset was created using the transformation pipeline defined in the ProblemGenerationAgent repository:

1. **Loading**: GAIR/LIMA dataset loaded from HuggingFace Hub
2. **Transformation**: Message-level transformations applied (HTML removal, whitespace normalization)
3. **Filtering**: Conversations filtered based on quality criteria:
- Minimum conversation length: 2 messages
- Minimum answer length: 50 characters
- Maximum answer length: 5000 characters
4. **Output**: Transformed conversations in message format

See `transformation_report.json` for detailed statistics about the transformation process.

See `rejection_log.json` for conversations that did not pass filtering criteria.

## Usage

```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("essobi/trl_gair_lima")

# Access examples
for example in dataset["train"]:
messages = example["messages"]
metadata = example["metadata"]
```

## License

This dataset follows the same license as the original GAIR/LIMA dataset: CC-BY-NC-4.0

## Citation

Original GAIR/LIMA dataset:
```
@article{zhou2023lima,
title={LIMA: Less Is More for Alignment},
author={Zhou, Chunting and Liu, Pengfei and Xu, Puxin and others},
journal={arXiv preprint arXiv:2305.11206},
year={2023}
}
```

## Disclaimer

This transformed dataset is provided as-is for research and educational purposes. The original LIMA dataset is subject to its own license and terms of use.

数据集信息：
特征：
- 名称：messages
列表类型：
- 名称：role
数据类型：字符串
- 名称：content
数据类型：字符串
- 名称：metadata
数据类型：JSON
划分集：
- 名称：train
- 名称：test
下载大小：未知
数据集大小：未知
许可证：CC-BY-NC-4.0

# GAIR/LIMA TRL 数据集
本数据集是[GAIR/LIMA](https://huggingface.co/datasets/GAIR/LIMA)数据集的转换版本，专为适配TRL（Transformers强化学习）训练而格式化。

## 数据集说明
- **主页**：[GAIR/LIMA](https://huggingface.co/datasets/GAIR/LIMA)
- **代码仓库**：[ProblemGenerationAgent](https://github.com/er-ads/ProblemGenerationAgent)
- **相关论文**：[LIMA：少即是多的对齐方法](https://arxiv.org/abs/2305.11206)

## 数据集概览
本数据集包含GAIR/LIMA数据集中的对话数据，已转换为适用于对话式AI训练与强化学习的基于消息的格式。

### 关键转换操作
- 从问答格式转换为多轮对话格式
- 执行消息级转换（移除HTML标签、标准化空白字符）
- 基于质量标准对对话进行过滤（长度检查、内容验证）
- 保留原始元数据以供参考与溯源

## 数据集结构
每个样本包含：
- `messages`：对话轮次列表，每个轮次包含：
- `role`：取值为"user"（用户）或"assistant"（助手）
- `content`：消息文本
- `metadata`：原始元数据与衍生元数据，包括：
- `original_id`：原始LIMA数据集中的ID
- `answers_count`：原始格式下的回答数量
- `original_data`：保留的原始字段

## 划分集
- **train**：训练划分集（经过过滤的LIMA训练集）
- **test**：测试划分集（经过过滤的LIMA测试集）

## 数据集构建流程
本数据集通过ProblemGenerationAgent代码仓库中定义的转换流水线构建：
1. **加载**：从HuggingFace Hub加载GAIR/LIMA数据集
2. **转换**：执行消息级转换（移除HTML标签、标准化空白字符）
3. **过滤**：基于质量标准对对话进行过滤：
- 最小对话长度：2个消息轮次
- 最小回答长度：50个字符
- 最大回答长度：5000个字符
4. **输出**：将转换后的对话保存为消息格式

可查看`transformation_report.json`以获取转换过程的详细统计数据。
可查看`rejection_log.json`以获取未通过过滤标准的对话记录。

## 使用示例
python
from datasets import load_dataset

# 加载数据集
dataset = load_dataset("essobi/trl_gair_lima")

# 访问样本
for example in dataset["train"]:
messages = example["messages"]
metadata = example["metadata"]

## 许可证
本数据集遵循原始GAIR/LIMA数据集的许可证：CC-BY-NC-4.0

## 引用信息
原始GAIR/LIMA数据集：

@article{zhou2023lima,
title={LIMA: Less Is More for Alignment},
author={Zhou, Chunting and Liu, Pengfei and Xu, Puxin and others},
journal={arXiv preprint arXiv:2305.11206},
year={2023}
}

## 免责声明
本转换数据集仅按现状提供，仅供研究与教育用途。原始LIMA数据集受其自身许可证与使用条款约束。