Yuri136/arxiv-papers

---
license: mit
language:
- en
size_categories:
- 1T<n<10T
task_categories:
- text-to-image
- visual-question-answering
- document-question-answering
- text-generation
---

# Complete ArXiv Papers Dataset (4.68 TB)

## 📚 Dataset Overview

This repository contains **the complete ArXiv scientific papers archive** organized by subject categories and publication years. With 4.68 TB of compressed PDFs and metadata, this represents one of the largest collections of scientific literature available for research and AI training.

## 🗂️ Dataset Structure

### Organized by Subject Categories:
- **astro-ph** (00-22): Astrophysics
- **cond-mat** (00-32): Condensed Matter Physics
- **cs** (00-62): Computer Science (most extensive category)
- **math** (00-52): Mathematics
- **physics** (00-16): General Physics
- **quant-ph** (00-12): Quantum Physics
- **stat** (00-05): Statistics
- **econ, eess, hep, nlin, q-bio, q-fin**: Specialized categories
- Plus additional specialized domains

### File Organization:
- Each category split into numbered segments (00, 01, 02...)
- Large categories further divided into parts (part-1, part-2, etc.)
- All files in ZIP format containing PDFs

## 📊 Dataset Statistics

- **Total Size**: 4.68 TB (compressed)
- **Format**: ZIP archives containing PDFs + metadata
- **Coverage**: Complete ArXiv historical archive
- **Organization**: By subject category

## 🎯 Primary Use Cases

### Multi-Modal AI Training
- **Scientific Document Understanding**: Train models on full PDF content
- **Figure-Caption Alignment**: Extract and pair scientific figures with their descriptions
- **Mathematical Reasoning**: Process complex mathematical notation and derivations
- **Cross-modal Retrieval**: Link textual concepts with visual scientific content

### Research Applications
- **Bibliometric Analysis**: Track research trends across decades
- **Scientific NLP**: Train domain-specific language models
- **Knowledge Extraction**: Parse algorithms, methodologies, and results
- **Academic Search**: Build enhanced scientific search engines

## 🛠️ Usage Examples

### Accessing Specific Categories
```python
# Example: Access Computer Science papers from segment 00
from huggingface_hub import hf_hub_download
import zipfile

file_path = hf_hub_download(
repo_id="nick007x/xiv-papers",
filename="cs-00.zip"
)

# Extract and process PDFs
with zipfile.ZipFile(file_path, 'r') as zip_ref:
zip_ref.extractall("cs_papers/")
```

### Working with Metadata
The `train.parquet` file contains structured metadata including:
- arXiv IDs, titles, authors, submission dates
- Abstracts, comments, primary subjects
- File paths to corresponding PDFs

## ⚡ Quick Start

1. **Browse Categories**: Start with smaller categories like `gr-qc-00.zip` (4.07 GB)
2. **Extract Metadata**: Use `train.parquet` for paper discovery
3. **Targeted Download**: Download specific subject areas of interest
4. **Stream Processing**: Handle large files with streaming extraction

## 🌟 Value Proposition

This dataset enables:
- **Complete Scientific Coverage**: Every paper from ArXiv's history
- **Multi-Domain Expertise**: Physics, CS, Math, Statistics, and more
- **Ready for Foundation Models**: Perfect for training scientific AI
- **Structured Organization**: Easy access by domain and time period

## 📜 License & Usage Terms

**Important:** This dataset is a **collection** of individual scholarly works from arXiv.org. The licensing structure is as follows:

* **The Collection (Metadata & Packaging):** The script used to create this dataset, the unique metadata (e.g., file structure, dataset description), and its packaging are licensed under the **MIT License**.
* **The Individual Papers (Content):** Each paper (PDF/TeX source) remains under the copyright and license chosen by its respective author(s). These licenses are typically Creative Commons (e.g., CC BY, CC BY-NC, CC BY-NC-ND).
* **Your Responsibility:** Users of this dataset are **solely responsible** for checking, understanding, and complying with the specific license terms of any paper they access, download, or use from this collection. You must provide appropriate attribution to the original authors as required by their chosen license.

**By using this dataset, you agree to these terms and acknowledge that the dataset creator is not liable for any license violations resulting from your use of the contained papers.**

For more information, see:
* [arXiv's Terms of Use](https://info.arxiv.org/help/license/index.html)
* [Creative Commons Licenses](https://creativecommons.org/licenses/)

## 🙏 Acknowledgments

This dataset builds upon the incredible work of:
- **ArXiv** team and moderators
- **Paper authors** across all scientific domains
- **Open scientific community** enabling knowledge sharing

---

**Note**: Due to the massive size (4.68 TB), consider downloading specific categories of interest rather than the entire dataset. The organized structure makes targeted access straightforward.

---
许可证：MIT协议（MIT License）
语言：英语
规模类别：1 TB < 数据集规模 < 10 TB
任务类别：
- 文本到图像生成（text-to-image）
- 视觉问答（visual-question-answering）
- 文档问答（document-question-answering）
- 文本生成（text-generation）
---

## 📚 数据集概览

本仓库收录**完整的ArXiv（arXiv）科学论文档案库**，按学科类别与发表年份进行组织。该数据集包含4.68 TB的压缩PDF文件与元数据，是当前可用于科研与人工智能训练的规模最大的科学文献馆藏之一。

## 🗂️ 数据集结构

### 按学科类别组织：
- **astro-ph（00-22）**：天体物理学
- **cond-mat（00-32）**：凝聚态物理
- **cs（00-62）**：计算机科学（规模最大的类别）
- **math（00-52）**：数学
- **physics（00-16）**：普通物理学
- **quant-ph（00-12）**：量子物理学
- **stat（00-05）**：统计学
- **econ、eess、hep、nlin、q-bio、q-fin**：其他专业学科类别
- 此外还包含更多细分领域

### 文件组织方式：
- 每个学科类别被拆分为编号分段（00、01、02……）
- 大型类别会进一步划分为多个子部分（part-1、part-2等）
- 所有文件均为ZIP压缩格式，内含PDF文件

## 📊 数据集统计信息

- **总规模**：4.68 TB（压缩后）
- **文件格式**：内含PDF与元数据的ZIP压缩包
- **覆盖范围**：完整的ArXiv历史档案
- **组织方式**：按学科类别分类

## 🎯 主要应用场景

### 多模态人工智能训练
- **科学文档理解**：基于完整PDF内容训练模型
- **图表-标题对齐**：提取科学图表并与其描述文本进行配对
- **数学推理**：处理复杂的数学符号与推导过程
- **跨模态检索**：将文本概念与可视化科学内容进行关联

### 科研应用
- **文献计量分析**：追踪数十年间的研究趋势
- **科学自然语言处理**：训练领域专属语言模型
- **知识抽取**：解析算法、研究方法与实验结果
- **学术搜索**：构建增强型科学搜索引擎

## 🛠️ 使用示例

### 访问特定学科类别
python
# 示例：访问计算机科学领域的00分段论文
from huggingface_hub import hf_hub_download
import zipfile

file_path = hf_hub_download(
repo_id="nick007x/xiv-papers",
filename="cs-00.zip"
)

# 解压并处理PDF文件
with zipfile.ZipFile(file_path, 'r') as zip_ref:
zip_ref.extractall("cs_papers/")

### 处理元数据
`train.parquet` 文件包含结构化元数据，具体包括：
- arXiv编号、论文标题、作者、提交日期
- 论文摘要、备注信息、主要学科分类
- 对应PDF文件的存储路径

## ⚡ 快速入门指南
1. **浏览学科类别**：可从较小的类别开始，例如`gr-qc-00.zip`（大小为4.07 GB）
2. **提取元数据**：使用`train.parquet`文件进行论文检索
3. **定向下载**：按需下载特定学科领域的数据
4. **流式处理**：采用流式解压方式处理大型文件

## 🌟 数据集价值优势

本数据集可支持以下应用：
- **完整的科学覆盖**：收录ArXiv历史上的全部论文
- **多领域专业覆盖**：涵盖物理学、计算机科学、数学、统计学等多个学科
- **适配基础模型训练**：非常适合用于训练科学领域人工智能模型
- **结构化组织形式**：可按学科领域与时间周期轻松获取数据

## 📜 许可证与使用条款

**重要说明**：本数据集是从arXiv.org收集的学术作品合集，其授权结构如下：
- **数据集合集（元数据与打包格式）**：用于构建本数据集的脚本、专属元数据（如文件结构、数据集描述）及其打包形式均采用**MIT许可证（MIT License）**进行授权。
- **单篇论文（内容本身）**：每篇论文（PDF/TeX源文件）的版权与授权仍由其原作者保留，通常采用知识共享（Creative Commons）协议（例如CC BY、CC BY-NC、CC BY-NC-ND）。
- **用户责任**：本数据集的使用者需**自行负责检查、理解并遵守所使用的每篇论文的具体授权条款**，并需按照原作者选定的授权协议要求，为原作者提供恰当的署名。

**使用本数据集即代表您同意上述条款，并确认数据集创建者不对因您使用其中论文所导致的任何授权违规行为承担责任。**

更多信息请参阅：
* [ArXiv使用条款](https://info.arxiv.org/help/license/index.html)
* [知识共享许可协议](https://creativecommons.org/licenses/)

## 🙏 致谢

本数据集基于以下团队与个人的卓越工作构建：
- **ArXiv** 团队与审核专员
- 所有科学领域的**论文作者**
- 推动知识共享的**开放科学社区**

---

**注意**：由于本数据集规模庞大（4.68 TB），建议按需下载特定学科类别而非整个数据集。其结构化的组织形式可让定向获取数据变得简单便捷。