Zzzz2323/structured-wikipedia

这是一个预解析的英文和法文维基百科文章数据集，使用Wikimedia Enterprise快照API提取。该数据集包含英文和法文维基百科的所有文章，预解析后以结构化数据形式输出，采用一致的架构。数据集以Parquet格式提供，优化了高性能分析查询和高效存储。此版本使用统一的固定架构，兼容DuckDB、pandas、Polars和Apache Spark。新特性包括：解析的参考文献和引用，连接维基百科知识与其来源；解析的表格，这是维基百科页面信息最密集的部分之一；可信度信号（如referenceneed和referencerisk），指示信息可能未充分依据来源；列表解析的改进，包括嵌套列表、有序列表和定义列表；以及文章正文图像已解析并可在文章正文部分有效载荷中使用。

Pre-parsed English and French Wikipedia articles, extracted using the Wikimedia Enterprise Snapshot API. This dataset contains all articles of the English and French language editions of Wikipedia, pre-parsed and output as structured data with a consistent schema. The dataset is provided in Parquet format, optimized for high-performance analytical queries and efficient storage. This version uses a unified, pinned schema across all files, making it compatible with DuckDB, pandas, Polars, and Apache Spark out of the box. New in this dataset: Parsed references and citations, connecting Wikipedias knowledge with its sources of truth. Parsed tables, one of the most information-heavy sections of Wikipedia pages. Credibility signals, for example `referenceneed` and `referencerisk`, signaling where information may not be sufficiently backed by sources. Improvements to list parsing, including nested lists, ordered lists, and definition lists. Article-body images are parsed and available in the article body section payload.