wyzard-ai/Adhaar193

--- size_categories: n<1K tags: - rlfh - argilla - human-feedback --- # Dataset Card for Adhaar193 This dataset has been created with [Argilla](https://github.com/argilla-io/argilla). As shown in the sections below, this dataset can be loaded into your Argilla server as explained in [Load with Argilla](#load-with-argilla), or used directly with the `datasets` library in [Load with `datasets`](#load-with-datasets). ## Using this dataset with Argilla To load with Argilla, you'll just need to install Argilla as `pip install argilla --upgrade` and then use the following code: ```python import argilla as rg ds = rg.Dataset.from_hub("wyzard-ai/Adhaar193", settings="auto") ``` This will load the settings and records from the dataset repository and push them to you Argilla server for exploration and annotation. ## Using this dataset with `datasets` To load the records of this dataset with `datasets`, you'll just need to install `datasets` as `pip install datasets --upgrade` and then use the following code: ```python from datasets import load_dataset ds = load_dataset("wyzard-ai/Adhaar193") ``` This will only load the records of the dataset, but not the Argilla settings. ## Dataset Structure This dataset repo contains: * Dataset records in a format compatible with HuggingFace `datasets`. These records will be loaded automatically when using `rg.Dataset.from_hub` and can be loaded independently using the `datasets` library via `load_dataset`. * The [annotation guidelines](#annotation-guidelines) that have been used for building and curating the dataset, if they've been defined in Argilla. * A dataset configuration folder conforming to the Argilla dataset format in `.argilla`. The dataset is created in Argilla with: **fields**, **questions**, **suggestions**, **metadata**, **vectors**, and **guidelines**. ### Fields The **fields** are the features or text of a dataset's records. For example, the 'text' column of a text classification dataset of the 'prompt' column of an instruction following dataset. | Field Name | Title | Type | Required | Markdown | | ---------- | ----- | ---- | -------- | -------- | | instruction | User instruction | text | True | True | ### Questions The **questions** are the questions that will be asked to the annotators. They can be of different types, such as rating, text, label_selection, multi_label_selection, or ranking. | Question Name | Title | Type | Required | Description | Values/Labels | | ------------- | ----- | ---- | -------- | ----------- | ------------- | | relevance_score | How Relevant is the conversation based upon expert. Is the conversation highly curated for you or not. Please don't judge accuracy. | rating | True | N/A | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | accuracy_score | How accurate is the conversation based upon persona | rating | True | if | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | clarity_score | How clear is the conversation based upon persona | rating | True | Is the LLM getting confused | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | actionable_score | How actionable is the conversation based upon persona | rating | True | Is the LLM response to actionable for example, it shows comparison card on the right question. | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | engagement_score | How engaging is the conversation based upon persona | rating | True | Are there a lot of question that are being shown if yes, high score else low score | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | completeness_score | is the conversation complete based upon persona | rating | True | is the conversation complete based upon persona, not leaving any key aspect out | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | feedback | feedback | text | True | What do you think can be improved in the given conversation. How good was the conversation as per you? | N/A | <!-- check length of metadata properties --> ### Metadata The **metadata** is a dictionary that can be used to provide additional information about the dataset record. | Metadata Name | Title | Type | Values | Visible for Annotators | | ------------- | ----- | ---- | ------ | ---------------------- | | conv_id | Conversation ID | | - | True | | turn | Conversation Turn | | 0 - 100 | True | ### Data Instances An example of a dataset instance in Argilla looks as follows: ```json { "_server_id": "40c0fe4a-a3a6-4c59-ad97-5aa4a5d828c4", "fields": { "instruction": "**user**: Hi Sofia\n**assistant**: Hello Adhaar! How can I assist you today? Are you looking for insights on any specific AI-driven software or exclusive deals in sales tools?" }, "id": "13aa1e20-4646-401b-9385-ce05a4164741", "metadata": { "conv_id": "fc8ca0a3-a0fd-43dc-9a2e-7f7aa9c927c4", "turn": 0 }, "responses": { "accuracy_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 10 } ], "actionable_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "clarity_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "completeness_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "engagement_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "feedback": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": "good" } ], "relevance_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 10 } ] }, "status": "completed", "suggestions": {}, "vectors": {} } ``` While the same record in HuggingFace `datasets` looks as follows: ```json { "_server_id": "40c0fe4a-a3a6-4c59-ad97-5aa4a5d828c4", "accuracy_score.responses": [ 10 ], "accuracy_score.responses.status": [ "submitted" ], "accuracy_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "actionable_score.responses": [ 9 ], "actionable_score.responses.status": [ "submitted" ], "actionable_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "clarity_score.responses": [ 9 ], "clarity_score.responses.status": [ "submitted" ], "clarity_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "completeness_score.responses": [ 9 ], "completeness_score.responses.status": [ "submitted" ], "completeness_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "conv_id": "fc8ca0a3-a0fd-43dc-9a2e-7f7aa9c927c4", "engagement_score.responses": [ 9 ], "engagement_score.responses.status": [ "submitted" ], "engagement_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "feedback.responses": [ "good" ], "feedback.responses.status": [ "submitted" ], "feedback.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "id": "13aa1e20-4646-401b-9385-ce05a4164741", "instruction": "**user**: Hi Sofia\n**assistant**: Hello Adhaar! How can I assist you today? Are you looking for insights on any specific AI-driven software or exclusive deals in sales tools?", "relevance_score.responses": [ 10 ], "relevance_score.responses.status": [ "submitted" ], "relevance_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "status": "completed", "turn": 0 } ``` ### Data Splits The dataset contains a single split, which is `train`. ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization [More Information Needed] #### Who are the source language producers? [More Information Needed] ### Annotations #### Annotation guidelines Review the user interactions with the chatbot. #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators [More Information Needed] ### Licensing Information [More Information Needed] ### Citation Information [More Information Needed] ### Contributions [More Information Needed]size_categories: 样本量小于1000 tags: - 基于人类反馈的强化学习（RLHF, Reinforcement Learning from Human Feedback） - Argilla - 人类反馈 # 《Adhaar193数据集卡片》 本数据集基于[Argilla](https://github.com/argilla-io/argilla)构建。如下文所述，该数据集既可按照[使用Argilla加载](#load-with-argilla)的说明导入至你的Argilla服务器，也可通过[使用数据集库加载](#load-with-datasets)中的`datasets`库直接使用。 ## 使用该数据集配合Argilla 要使用Argilla加载，只需先通过`pip install argilla --upgrade`升级安装Argilla，随后运行如下代码： python import argilla as rg ds = rg.Dataset.from_hub("wyzard-ai/Adhaar193", settings="auto") 该代码将从数据集仓库加载配置与记录，并推送至你的Argilla服务器以供探索与标注。 ## 使用该数据集配合`datasets`库 要使用`datasets`库加载该数据集的记录，只需先通过`pip install datasets --upgrade`升级安装数据集库，随后运行如下代码： python from datasets import load_dataset ds = load_dataset("wyzard-ai/Adhaar193") 该代码仅会加载数据集的记录，而非Argilla相关配置。 ## 数据集结构 本数据集仓库包含： * 兼容Hugging Face `datasets`库格式的数据集记录。使用`rg.Dataset.from_hub`时会自动加载这些记录，也可通过`datasets`库的`load_dataset`函数独立加载。 * 用于构建与整理该数据集的[标注指南](#annotation-guidelines)（若已在Argilla中定义）。 * 符合Argilla数据集格式的`.argilla`数据集配置文件夹。 本数据集在Argilla中通过**字段（fields）**、**问题（questions）**、**建议（suggestions）**、**元数据（metadata）**、**向量（vectors）**与**指南（guidelines）**构建。 ### 字段（Fields） **字段**即数据集记录的特征或文本内容。例如文本分类数据集的`text`列，或指令跟随数据集的`prompt`列。 | 字段名 | 字段标题 | 类型 | 是否必填 | 是否支持Markdown | | ------------ | -------------- | ---- | -------- | ---------------- | | instruction | 用户指令 | text | 是 | 是 | ### 问题（Questions） **问题**即向标注者提出的调研问题，支持评分、文本、标签选择、多标签选择与排序等多种类型。 | 问题名 | 问题标题 | 类型 | 是否必填 | 问题描述 | 取值/标签 | | ------------------ | ------------------------------------------------------------------------ | ------ | -------- | ------------------------------------------------------------------------ | ----------------------------- | | relevance_score | 基于专家视角，该对话的相关程度如何？该对话是否为你精心整理？请勿评判准确性。 | rating | 是 | 无适用说明 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | accuracy_score | 基于角色设定，该对话的准确性如何？ | rating | 是 | 无适用说明 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | clarity_score | 基于角色设定，该对话的清晰度如何？ | rating | 是 | 大语言模型（LLM, Large Language Model）是否出现困惑 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | actionable_score | 基于角色设定，该对话的可操作性如何？ | rating | 是 | 大语言模型的回复是否具备可操作性，例如针对对应问题展示对比卡片 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | engagement_score | 基于角色设定，该对话的互动性如何？ | rating | 是 | 若展示了大量问题则得高分，反之则得低分 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | completeness_score | 基于角色设定，该对话是否完整？ | rating | 是 | 该对话是否完整，未遗漏任何关键要点 | [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | | feedback | 反馈意见 | text | 是 | 你认为该对话可在哪些方面改进？依你之见，该对话表现如何？ | 无适用说明 | ### 元数据（Metadata） **元数据**为可用于提供数据集记录额外信息的字典结构。 | 元数据名 | 元数据标题 | 类型 | 取值范围 | 是否对标注者可见 | | -------- | ---------------- | ---- | -------------- | ---------------- | | conv_id | 对话ID | | 无指定取值 | 是 | | turn | 对话轮次 | | 0 - 100 | 是 | ### 数据实例 Argilla中的数据集示例如以下JSON格式所示： json { "_server_id": "40c0fe4a-a3a6-4c59-ad97-5aa4a5d828c4", "fields": { "instruction": "**user**: Hi Sofia **assistant**: Hello Adhaar! How can I assist you today? Are you looking for insights on any specific AI-driven software or exclusive deals in sales tools?" }, "id": "13aa1e20-4646-401b-9385-ce05a4164741", "metadata": { "conv_id": "fc8ca0a3-a0fd-43dc-9a2e-7f7aa9c927c4", "turn": 0 }, "responses": { "accuracy_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 10 } ], "actionable_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "clarity_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "completeness_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "engagement_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 9 } ], "feedback": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": "good" } ], "relevance_score": [ { "user_id": "de1eeab4-62ca-4354-8a2c-f9454a59131e", "value": 10 } ] }, "status": "completed", "suggestions": {}, "vectors": {} } 而Hugging Face `datasets`库中的同一条记录格式如下： json { "_server_id": "40c0fe4a-a3a6-4c59-ad97-5aa4a5d828c4", "accuracy_score.responses": [ 10 ], "accuracy_score.responses.status": [ "submitted" ], "accuracy_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "actionable_score.responses": [ 9 ], "actionable_score.responses.status": [ "submitted" ], "actionable_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "clarity_score.responses": [ 9 ], "clarity_score.responses.status": [ "submitted" ], "clarity_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "completeness_score.responses": [ 9 ], "completeness_score.responses.status": [ "submitted" ], "completeness_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "conv_id": "fc8ca0a3-a0fd-43dc-9a2e-7f7aa9c927c4", "engagement_score.responses": [ 9 ], "engagement_score.responses.status": [ "submitted" ], "engagement_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "feedback.responses": [ "good" ], "feedback.responses.status": [ "submitted" ], "feedback.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "id": "13aa1e20-4646-401b-9385-ce05a4164741", "instruction": "**user**: Hi Sofia **assistant**: Hello Adhaar! How can I assist you today? Are you looking for insights on any specific AI-driven software or exclusive deals in sales tools?", "relevance_score.responses": [ 10 ], "relevance_score.responses.status": [ "submitted" ], "relevance_score.responses.users": [ "de1eeab4-62ca-4354-8a2c-f9454a59131e" ], "status": "completed", "turn": 0 } ### 数据划分 本数据集仅包含一个划分：`train`（训练集）。 ## 数据集创建 ### 整理依据 [需补充更多信息] ### 源数据 #### 初始数据收集与标准化 [需补充更多信息] #### 源语言生产者是谁？ [需补充更多信息] ### 标注 #### 标注指南 审阅与聊天机器人的用户交互内容。 #### 标注流程 [需补充更多信息] #### 标注者是谁？ [需补充更多信息] ### 个人与敏感信息 [需补充更多信息] ## 数据使用注意事项 ### 数据集的社会影响 [需补充更多信息] ### 偏差讨论 [需补充更多信息] ### 其他已知局限 [需补充更多信息] ## 附加信息 ### 数据集整理者 [需补充更多信息] ### 授权信息 [需补充更多信息] ### 引用信息 [需补充更多信息] ### 贡献 [需补充更多信息]