PeytonT/2m_papers_text

该数据集合并了现有的论文文本parquet导出文件与一个或多个回填parquet分片，然后每个canonical_paper_id保留一行。每行对应一篇规范的arXiv论文。当有多个版本可用时，合并过程根据仓库的选择逻辑保留最强的文本行：优先非部分文本，然后优先原始PDF派生的文本，然后较新版本，然后较长的文本。

This dataset merges an existing paper-text parquet export with one or more backfill parquet shards, then keeps exactly one row per `canonical_paper_id`. One row corresponds to one canonical arXiv paper. Where multiple versions are available, the merge keeps the strongest text row according to the repos selection logic: prefer non-partial text, then prefer raw-PDF-derived text, then newer versions, then longer text.