open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean

---
pretty_name: Evaluation run of quantumaikr/llama-2-70b-fb16-korean
dataset_summary: "Dataset automatically created during the evaluation run of model\
\ [quantumaikr/llama-2-70b-fb16-korean](https://huggingface.co/quantumaikr/llama-2-70b-fb16-korean)\
\ on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n\
\nThe dataset is composed of 3 configuration, each one coresponding to one of the\
\ evaluated task.\n\nThe dataset has been created from 1 run(s). Each run can be\
\ found as a specific split in each configuration, the split being named using the\
\ timestamp of the run.The \"train\" split is always pointing to the latest results.\n\
\nAn additional configuration \"results\" store all the aggregated results of the\
\ run (and is used to compute and display the agregated metrics on the [Open LLM\
\ Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)).\n\
\nTo load the details from a run, you can for instance do the following:\n```python\n\
from datasets import load_dataset\ndata = load_dataset(\"open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean\"\
,\n\t\"harness_winogrande_5\",\n\tsplit=\"train\")\n```\n\n## Latest results\n\n\
These are the [latest results from run 2023-10-17T08:56:24.573395](https://huggingface.co/datasets/open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean/blob/main/results_2023-10-17T08-56-24.573395.json)(note\
\ that their might be results for other tasks in the repos if successive evals didn't\
\ cover the same tasks. You find each in the results and the \"latest\" split for\
\ each eval):\n\n```python\n{\n \"all\": {\n \"em\": 0.0041946308724832215,\n\
\ \"em_stderr\": 0.0006618716168266237,\n \"f1\": 0.07418729026845645,\n\
\ \"f1_stderr\": 0.0015820737575191846,\n \"acc\": 0.5583664886878857,\n\
\ \"acc_stderr\": 0.011574854481074981\n },\n \"harness|drop|3\": {\n\
\ \"em\": 0.0041946308724832215,\n \"em_stderr\": 0.0006618716168266237,\n\
\ \"f1\": 0.07418729026845645,\n \"f1_stderr\": 0.0015820737575191846\n\
\ },\n \"harness|gsm8k|5\": {\n \"acc\": 0.29037149355572406,\n \
\ \"acc_stderr\": 0.012503592481818962\n },\n \"harness|winogrande|5\"\
: {\n \"acc\": 0.8263614838200474,\n \"acc_stderr\": 0.010646116480331\n\
\ }\n}\n```"
repo_url: https://huggingface.co/quantumaikr/llama-2-70b-fb16-korean
leaderboard_url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
point_of_contact: [e-mail]
configs:
- config_name: harness_drop_3
data_files:
- split: 2023_10_17T08_56_24.573395
path:
- '**/details_harness|drop|3_2023-10-17T08-56-24.573395.parquet'
- split: latest
path:
- '**/details_harness|drop|3_2023-10-17T08-56-24.573395.parquet'
- config_name: harness_gsm8k_5
data_files:
- split: 2023_10_17T08_56_24.573395
path:
- '**/details_harness|gsm8k|5_2023-10-17T08-56-24.573395.parquet'
- split: latest
path:
- '**/details_harness|gsm8k|5_2023-10-17T08-56-24.573395.parquet'
- config_name: harness_winogrande_5
data_files:
- split: 2023_10_17T08_56_24.573395
path:
- '**/details_harness|winogrande|5_2023-10-17T08-56-24.573395.parquet'
- split: latest
path:
- '**/details_harness|winogrande|5_2023-10-17T08-56-24.573395.parquet'
- config_name: results
data_files:
- split: 2023_10_17T08_56_24.573395
path:
- results_2023-10-17T08-56-24.573395.parquet
- split: latest
path:
- results_2023-10-17T08-56-24.573395.parquet
---

# Dataset Card for Evaluation run of quantumaikr/llama-2-70b-fb16-korean

## Dataset Description

- **Homepage:**
- **Repository:** https://huggingface.co/quantumaikr/llama-2-70b-fb16-korean
- **Paper:**
- **Leaderboard:** https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- **Point of Contact:** [e-mail]

### Dataset Summary

Dataset automatically created during the evaluation run of model [quantumaikr/llama-2-70b-fb16-korean](https://huggingface.co/quantumaikr/llama-2-70b-fb16-korean) on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).

The dataset is composed of 3 configuration, each one coresponding to one of the evaluated task.

The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.

An additional configuration "results" store all the aggregated results of the run (and is used to compute and display the agregated metrics on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)).

To load the details from a run, you can for instance do the following:
```python
from datasets import load_dataset
data = load_dataset("open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean",
"harness_winogrande_5",
split="train")
```

## Latest results

These are the [latest results from run 2023-10-17T08:56:24.573395](https://huggingface.co/datasets/open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean/blob/main/results_2023-10-17T08-56-24.573395.json)(note that their might be results for other tasks in the repos if successive evals didn't cover the same tasks. You find each in the results and the "latest" split for each eval):

```python
{
"all": {
"em": 0.0041946308724832215,
"em_stderr": 0.0006618716168266237,
"f1": 0.07418729026845645,
"f1_stderr": 0.0015820737575191846,
"acc": 0.5583664886878857,
"acc_stderr": 0.011574854481074981
},
"harness|drop|3": {
"em": 0.0041946308724832215,
"em_stderr": 0.0006618716168266237,
"f1": 0.07418729026845645,
"f1_stderr": 0.0015820737575191846
},
"harness|gsm8k|5": {
"acc": 0.29037149355572406,
"acc_stderr": 0.012503592481818962
},
"harness|winogrande|5": {
"acc": 0.8263614838200474,
"acc_stderr": 0.010646116480331
}
}
```

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

[More Information Needed]

## Dataset Structure

### Data Instances

[More Information Needed]

### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]

原始信息汇总

数据集卡片 for Evaluation run of quantumaikr/llama-2-70b-fb16-korean

数据集描述

数据集概述

该数据集是在模型 quantumaikr/llama-2-70b-fb16-korean 在 Open LLM Leaderboard 上的评估运行期间自动创建的。

数据集由3个配置组成，每个配置对应一个评估任务。

数据集从1次运行中创建。每次运行可以在每个配置中作为一个特定的分割找到，分割名称使用运行的时间戳。"train" 分割始终指向最新的结果。

一个额外的配置 "results" 存储了所有运行结果的聚合（用于计算和显示 Open LLM Leaderboard 上的聚合指标）。

要加载某个运行的详细信息，可以执行以下操作： python from datasets import load_dataset data = load_dataset("open-llm-leaderboard/details_quantumaikr__llama-2-70b-fb16-korean", "harness_winogrande_5", split="train")

最新结果

以下是 2023-10-17T08:56:24.573395 运行的最新结果（注意，如果连续评估没有覆盖相同的任务，仓库中可能会有其他任务的结果。您可以在 "results" 和每个评估的 "latest" 分割中找到每个任务的结果）：

python { "all": { "em": 0.0041946308724832215, "em_stderr": 0.0006618716168266237, "f1": 0.07418729026845645, "f1_stderr": 0.0015820737575191846, "acc": 0.5583664886878857, "acc_stderr": 0.011574854481074981 }, "harness|drop|3": { "em": 0.0041946308724832215, "em_stderr": 0.0006618716168266237, "f1": 0.07418729026845645, "f1_stderr": 0.0015820737575191846 }, "harness|gsm8k|5": { "acc": 0.29037149355572406, "acc_stderr": 0.012503592481818962 }, "harness|winogrande|5": { "acc": 0.8263614838200474, "acc_stderr": 0.010646116480331 } }

搜集汇总

数据集介绍

构建方式

在大语言模型评估领域，对模型性能的客观量化是推动技术进步的关键环节。该数据集专为记录quantumaikr/llama-2-70b-fb16-korean模型在Open LLM Leaderboard上的评估过程而自动生成。其构建方式基于单次评估运行，将模型在三个不同任务上的表现数据分别存储为独立配置（harness_drop_3、harness_gsm8k_5、harness_winogrande_5），每个配置内以运行时间戳命名数据分割，并设定'train'分割始终指向最新结果。此外，增设'results'配置用于汇总全部聚合指标，为排行榜的度量展示提供数据支撑。

特点

该数据集最显著的特点在于其自动化与结构化设计。它通过三个任务配置精确对应DROP、GSM8K和Winogrande评估基准，每个配置均包含完整的细粒度评估细节。数据分割采用时间戳命名策略，既保留了历史运行轨迹，又通过'train'分割确保最新结果的即时访问。'results'配置则进一步整合了精确匹配率、F1分数及准确率等关键指标及其标准误差，为模型性能的横向对比提供了标准化、可追溯的量化依据。

使用方法

研究人员可通过HuggingFace的datasets库便捷调用该数据集。具体而言，使用load_dataset函数加载指定配置（如'harness_winogrande_5'）并选择'train'分割即可获取最新评估详情。对于需要回溯历史运行的情况，可通过时间戳命名的分割访问特定批次数据。此外，'results'配置支持直接加载聚合后的综合指标，便于进行快速性能概览与多模型对比分析，从而高效支撑大语言模型评估研究中的实证需求。

背景与挑战

背景概述

随着大语言模型在自然语言处理领域的迅猛发展，如何系统化、标准化地评估模型性能成为学术界与工业界共同关注的焦点。2023年，Hugging Face团队联合多方研究者推出了Open LLM Leaderboard，旨在为开源大语言模型提供透明、可复现的评测平台。该数据集正是对quantumaikr/llama-2-70b-fb16-korean模型在2023年10月17日进行的单次评测记录的完整存档，涵盖了DROP、GSM8K和WinoGrande三项代表性任务。作为韩国语增强版Llama-2-70B模型的评估数据，它不仅记录了模型在阅读理解、数学推理和常识推理上的具体表现，更反映了当前多语言大模型在非英语场景下的能力边界，对推动韩语自然语言处理研究具有重要参考价值。

当前挑战

该数据集所面临的挑战主要来源于大语言模型评测的固有难题与构建过程的实际困难。在领域问题层面，评测任务覆盖了需要深度理解与推理的DROP数据集、涉及多步数学推导的GSM8K以及依赖世界知识的WinoGrande，这些任务对模型的泛化能力和鲁棒性提出了极高要求，而该模型在DROP上的精确匹配率仅为0.42%，凸显了复杂阅读理解任务的艰巨性。在构建过程中，数据集仅基于单次评测运行生成，缺乏多次重复实验以评估结果的稳定性；同时，评测配置固定为少数几个样本量（如3-shot或5-shot），未能覆盖不同提示策略对性能的影响。此外，数据集的元数据未能提供模型训练数据的具体分布、微调细节或语言增强方法，限制了研究者对评测结果进行深入归因分析的能力。

常用场景

经典使用场景

在大型语言模型评估领域，Open LLM Leaderboard上的评估运行数据集为模型性能的标准化度量提供了重要基准。该数据集专门用于记录quantumaikr/llama-2-70b-fb16-korean模型在多项经典任务上的表现，涵盖DROP阅读理解、GSM8K数学推理以及WinoGrande常识推理等核心评估场景。研究者和开发者可通过加载该数据集，复现模型在特定任务上的详细得分，从而精准比较不同模型在统一评测框架下的能力差异，为模型选型与优化提供数据支撑。

实际应用

在实际应用中，该数据集为模型开发团队提供了便捷的性能监控与迭代反馈工具。当quantumaikr/llama-2-70b-fb16-korean等模型被部署于客服对话、教育辅导或文档分析等场景时，开发者可借助数据集中的详细评测结果，快速定位模型在特定任务上的薄弱环节，从而针对性地进行微调或适配。此外，数据集中的聚合指标还可用于向用户展示模型能力概况，增强产品可信度。

衍生相关工作

该评估运行数据集衍生了一系列关于多语言大模型能力对齐与跨任务泛化的经典研究工作。基于其提供的标准化评测结果，学术界进一步探索了韩语模型在通用推理任务上的表现与英语基座模型之间的迁移规律，催生了诸如“跨语言能力映射”、“少样本学习鲁棒性分析”等研究方向。同时，该数据集与Open LLM Leaderboard生态紧密结合，成为后续模型排行、评估方法论改进的重要数据来源，推动了更科学的模型评价体系构建。