NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset
收藏资源简介：
--- license: odc-by task_categories: - text-generation language: - en tags: - language-modeling - casual-lm - llm pretty_name: Dolma size_categories: - 100B<n<1T --- Tokenized (Llama 3) verison of [NousResearch/dolma-v1_7-305B](https://huggingface.co/datasets/NousResearch/dolma-v1_7-305B) as a [Nanotron](https://github.com/huggingface/nanotron) dataset split into 10 GB chunks. To download: ```shell huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset ``` To recombine: ```shell cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy rm -rf dolma-v1_7-305B-tokenized-llama3-nanoset ``` Can also be used directly with numpy, for example ```python import numpy as np dataset_buffer_mmap = np.memmap("dolma-v1_7-305B-tokenized-llama3-nanoset.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer)) ```
数据集概述
基本信息
- 许可证: odc-by
- 任务类别: text-generation
- 语言: en
- 标签:
- language-modeling
- casual-lm
- llm
- 名称: Dolma
- 大小类别: 100B<n<1T
数据集描述
- 版本: Dolma-v1_7-305B-tokenized-llama3-nanoset
- 格式: 分为10 GB的块，使用Nanotron格式
- 原始数据集: NousResearch/dolma-v1_7-305B
- 处理方式: 使用Llama 3进行Token化处理
使用方法
-
下载命令: shell huggingface-cli download --repo-type dataset --local-dir dolma-v1_7-305B-tokenized-llama3-nanoset --local-dir-use-symlinks False NousResearch/dolma-v1_7-305B-tokenized-llama3-nanoset
-
重组命令: shell cat dolma-v1_7-305B-tokenized-llama3-nanoset/dolma-v1_7-305B-tokenized-llama3-nanoset.npy.* > dolma-v1_7-305B-tokenized-llama3-nanoset.npy rm -rf dolma-v1_7-305B-tokenized-llama3-nanoset
-
直接使用示例: python import numpy as np
dataset_buffer_mmap = np.memmap("dolma-v1_7-305B-tokenized-llama3-nanoset.npy", mode="r", order="C", dtype=np.int32) dataset_buffer = memoryview(dataset_buffer_mmap) dataset_number_of_tokens = int(len(dataset_buffer))
