lance-format/fineweb-edu

FineWeb-Edu (Lance格式)是FineWeb-Edu数据集的Lance格式版本，包含超过15亿个教育类网页段落。每个段落包含清理后的文本、源URL、标题、语言检测信号（语言代码和置信度）以及预计算的384维文本嵌入。数据集采用Lance列式存储格式，支持高效随机访问、多模态数据存储、本地索引构建、数据演化和版本控制。适用于文本检索、问答、机器学习训练等任务，特别适合大规模教育内容分析和检索应用。数据来源于网络，遵循ODC-BY 1.0许可。

FineWeb-Edu (Lance Format) is a Lance-formatted version of the FineWeb-Edu dataset, containing over 1.5 billion educational web passages. Each passage includes cleaned text, source URL, title, language detection signals (language code and confidence), and pre-computed 384-dim text embeddings. The dataset uses the Lance columnar storage format, enabling efficient random access, multimodal data storage, local index building, data evolution, and versioning. It is suitable for text retrieval, question-answering, machine learning training, and other applications, particularly for large-scale educational content analysis and retrieval. The data is sourced from the web and distributed under the ODC-BY 1.0 license.