open-index/open-html
收藏资源简介：
--- license: odc-by task_categories: - text-generation - feature-extraction - text-classification language: - en - mul pretty_name: OpenHTML size_categories: - 1M<n<10M tags: - common-crawl - web-crawl - html - text - metadata configs: - config_name: default data_files: - split: train path: data/*/* - config_name: CC-MAIN-2026-12 data_files: - split: train path: data/CC-MAIN-2026-12/* --- # **OpenHTML** > Raw HTML from the web with rich structured metadata — ready for training, retrieval, and analysis ## What is it? **OpenHTML** is a large-scale web dataset built from [Common Crawl](https://commoncrawl.org). Common Crawl is a non-profit that crawls the web and freely provides its archives and datasets to the public — see [their latest crawl announcement](https://commoncrawl.org/blog/march-2026-crawl-archive-now-available) for details on the source data. Every page goes through a pipeline that extracts the raw HTML body along with structured metadata from WARC records, HTTP response headers, and HTML `<head>` tags, then packages everything into Parquet files with 24 columns. The dataset currently includes crawl **CC-MAIN-2026-12** with **197,357 documents across 10 shards**. Processed 34.3 GB of raw HTML into 34.3 GB of stored body text — 6.5 GB as Parquet (Zstd). We plan to add more snapshots over time. **OpenHTML** is released under the **Open Data Commons Attribution License (ODC-By) v1.0**, the same license used by Common Crawl. ## What is being released? Each Common Crawl WARC file (~1 GB of compressed HTML) becomes one Parquet shard. The shards live under a crawl-specific directory so multiple snapshots can coexist: ``` data/ CC-MAIN-2026-12/ 00000.parquet 00001.parquet ... ``` Every row in a Parquet file is one web page with **24 columns** of metadata. Each row includes the `warc_record_id` and `warc_date` fields parsed from the original WARC headers, so you can trace any document back to its source record. We also extract HTTP response headers (`content_type`, `charset`, `content_language`, `http_server`, `http_last_modified`) and HTML `<head>` metadata (`title`, `description`, `og:title`, `og:description`, `og:image`, `og:type`, `canonical_url`, `html_lang`). The URL is decomposed into `host`, `domain` (eTLD+1), `path`, and `query`. ## How to download and use OpenHTML ### Using `datasets` ```python from datasets import load_dataset # stream the entire dataset ds = load_dataset("open-index/open-html", name="CC-MAIN-2026-12", split="train", streaming=True) for doc in ds: print(doc["url"], doc["title"], len(doc["body"])) # load a single shard into memory ds = load_dataset( "open-index/open-html", data_files="data/CC-MAIN-2026-12/00000.parquet", split="train", ) ``` ### Using `huggingface_hub` ```python from huggingface_hub import snapshot_download folder = snapshot_download( "open-index/open-html", repo_type="dataset", local_dir="./open-html/", allow_patterns="data/CC-MAIN-2026-12/*", ) ``` For faster downloads, install `pip install huggingface_hub[hf_transfer]` and set `HF_HUB_ENABLE_HF_TRANSFER=1`. ### Using DuckDB ```sql SELECT url, title, domain, html_lang, html_length FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') WHERE domain = 'wikipedia.org' LIMIT 10; ``` ```sql -- Top domains by page count SELECT domain, COUNT(*) as pages, AVG(html_length) as avg_html FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') GROUP BY domain ORDER BY pages DESC LIMIT 20; ``` ```sql -- Pages with Open Graph metadata SELECT url, og_title, og_description, og_image FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') WHERE og_title != '' AND og_image != '' LIMIT 10; ``` # Dataset card for OpenHTML ## Dataset Description - **Homepage and Repository:** [https://huggingface.co/datasets/open-index/open-html](https://huggingface.co/datasets/open-index/open-html) - **Point of Contact:** please create a discussion on the Community tab - **License:** Open Data Commons Attribution License (ODC-By) v1.0 ## Dataset Structure ### Data Instance The following is an example row from the dataset: ```json { "url": "https://example.com/article/interesting-topic", "warc_date": "2026-03-05T07:14:58Z", "warc_record_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "warc_filename": "CC-MAIN-20260305070756-20260305100756-00000.warc.gz", "http_status": 200, "content_type": "text/html", "charset": "utf-8", "content_language": "en", "http_server": "nginx", "http_last_modified": "Tue, 04 Mar 2026 12:00:00 GMT", "host": "example.com", "domain": "example.com", "path": "/article/interesting-topic", "query": "", "html_lang": "en", "title": "Interesting Topic - Example", "description": "A fascinating article about interesting topics.", "og_title": "Interesting Topic", "og_description": "A fascinating article about interesting topics.", "og_image": "https://example.com/images/topic.jpg", "og_type": "article", "canonical_url": "https://example.com/article/interesting-topic", "html_length": 48210, "body": "<!DOCTYPE html><html lang=\"en\"><head>..." } ``` ### Data Fields | Column | Type | Description | |---|---|---| | `url` | string | Full URL of the crawled page | | `warc_date` | string | Crawl timestamp from the WARC record (RFC 3339) | | `warc_record_id` | string | UUID from the WARC-Record-ID header, for source traceability | | `warc_filename` | string | Source WARC file basename from Common Crawl | | `http_status` | int32 | HTTP response status code (always 200 in this dataset) | | `content_type` | string | Content-Type from the HTTP response (always starts with `text/html`) | | `charset` | string | Character encoding from the Content-Type header (e.g., `utf-8`, `iso-8859-1`) | | `content_language` | string | Content-Language HTTP header (e.g., `en`, `de`, `fr`) | | `http_server` | string | Server software from the HTTP response (e.g., `nginx`, `Apache`) | | `http_last_modified` | string | Last-Modified HTTP header — when the page was last changed | | `host` | string | Lowercase hostname extracted from the URL (e.g., `www.example.com`) | | `domain` | string | Registered domain (eTLD+1) — groups subdomains together (e.g., `example.com`) | | `path` | string | URL path component (e.g., `/article/interesting-topic`) | | `query` | string | URL query string, if any (e.g., `page=2&sort=date`) | | `html_lang` | string | Language attribute from `<html lang="...">` tag | | `title` | string | Page title from `<title>` tag in `<head>` | | `description` | string | Meta description from `<meta name="description">` | | `og_title` | string | Open Graph title from `<meta property="og:title">` | | `og_description` | string | Open Graph description from `<meta property="og:description">` | | `og_image` | string | Open Graph image URL from `<meta property="og:image">` | | `og_type` | string | Open Graph type from `<meta property="og:type">` (e.g., `article`, `website`) | | `canonical_url` | string | Canonical URL from `<link rel="canonical">` — the page's preferred URL | | `html_length` | int64 | Byte length of the raw HTML body in bytes | | `body` | string | Raw HTML body (full content, no truncation) | ### Data Splits The default subset includes all available data across all crawl snapshots. You can also load a specific crawl by using its ID as the config name (e.g. `CC-MAIN-2026-12`). ## Dataset Creation ### Curation Rationale Most open web datasets either release raw text (losing structure) or processed markdown (losing metadata). **OpenHTML** takes a different approach: it preserves the **raw HTML** alongside **24 columns of structured metadata** extracted from WARC headers, HTTP response headers, and HTML `<head>` tags. This lets you: - **Train** models on raw web content with full context - **Filter** by language, domain, content type, or Open Graph metadata - **Analyze** web structure, server software distribution, or charset usage - **Trace** every document back to its exact WARC source record ### Source Data The source data consists of web pages crawled by the [Common Crawl](https://commoncrawl.org) foundation. Common Crawl archives billions of pages across the public web and makes the raw WARC files freely available on Amazon S3. ### Data Processing Steps The processing pipeline runs as a single-pass extraction: 1. **Download** raw .warc.gz files from Common Crawl S3 (each file is roughly 1 GB compressed) 2. **Filter** to keep only HTTP 200 responses with a `text/html` content type, discarding images, scripts, redirects, and error pages 3. **Parse** HTTP response headers to extract `content_type`, `charset`, `content_language`, `server`, and `last_modified` 4. **Decompose** the URL into `host`, `domain` (eTLD+1 via the Public Suffix List), `path`, and `query` 5. **Extract** HTML `<head>` metadata using a streaming tokenizer: `title`, `description`, Open Graph tags (`og:title`, `og:description`, `og:image`, `og:type`), `canonical_url`, and `html_lang` 6. **Store** the full HTML body (no truncation — `html_length` matches `body` size) 7. **Export** directly to Apache Parquet with Zstd compression, 100,000 rows per row group No intermediate files are created — the pipeline streams from compressed WARC through extraction directly into Parquet. Pages that produce empty HTML bodies are dropped. ### Compression Ratios Numbers below are actual measurements summed across all 10 files of CC-MAIN-2026-12 (197,357 pages total), projected to the full crawl of 100,000 WARC files. | Stage | 10 files (measured) | 100,000 files (projected) | Reduction | |---|---|---|---| | Raw WARC (.warc.gz, downloaded) | ~8.1 GB | ~79.2 TB | — | | HTML extracted (uncompressed) | 34.3 GB | ~335.2 TB | — | | Body stored (full HTML) | 34.3 GB | ~335.2 TB | **-0.0%** vs HTML | | Final Parquet (Zstd) | 6.5 GB | ~63.8 TB | **-81.0%** vs body | The body column stores the full raw HTML. Parquet with Zstd then compresses the data further. End to end: ~8.1 GB of raw gzipped WARCs becomes **6.5 GB of Parquet** — a **19.5% total reduction** — containing 197,357 web pages with full metadata. ### Processing Times Pipeline timings across 10 shards of CC-MAIN-2026-12: ``` Download (raw WARC) ████████████████████████ 1h 29m 47s Extract (WARC → HTML + metadata) ███████████████████████░ 1h 28m 15s Publish (HuggingFace upload) ███░░░░░░░░░░░░░░░░░░░░░ 12m 58s ``` ### Dataset Charts   ### Personal and Sensitive Information No additional PII filtering is applied beyond what Common Crawl provides. As the dataset is sourced from the public web, it is likely that some personally identifiable information is present. If you find your own PII in the dataset and would like it removed, please open an issue on the repository. ## Considerations for Using the Data ### Social Impact By releasing both the dataset and the full processing pipeline, we aim to lower the barrier to training and evaluating language models on high quality web data. Researchers and practitioners who cannot afford to run their own Common Crawl processing pipelines can use **OpenHTML** directly. ### Discussion of Biases **OpenHTML** inherits the biases present in Common Crawl and the public web at large. The filtering step keeps only `text/html` pages, which may underrepresent content served as other content types. We have not applied any machine-learning-based quality or toxicity filters, as such filters have been shown to disproportionately remove content from certain dialects and communities. ### Known Limitations The full HTML body is stored without truncation. Very large pages (e.g., pages with inline data URIs) will increase shard sizes. The `html_length` field reflects the exact body size in bytes. Metadata extraction scans only the `<head>` section for performance. Pages that place `<meta>` or `<title>` tags in the `<body>` will have missing metadata. ## Additional Information ### Licensing The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0**. The use of this dataset is also subject to [Common Crawl's Terms of Use](https://commoncrawl.org/terms-of-use). The original content remains subject to the rights and terms of its respective publishers. ### Contact Please open a discussion on the [Community tab](https://huggingface.co/datasets/open-index/open-html/discussions) for questions, feedback, or issues.
--- 许可证: odc-by 任务类别: - 文本生成 - 特征提取 - 文本分类 语言: - en - mul 展示名称: OpenHTML 数据规模类别: - 1M<n<10M 标签: - common-crawl - web-crawl - html - text - metadata 配置: - 配置名称: default 数据文件: - 拆分: train 路径: data/*/* - 配置名称: CC-MAIN-2026-12 数据文件: - 拆分: train path: data/CC-MAIN-2026-12/* --- # **OpenHTML** > 源自互联网的原生HTML数据集，附带丰富的结构化元数据，可直接用于模型训练、信息检索与数据分析 ## 什么是OpenHTML？ **OpenHTML**是源自[通用爬虫（Common Crawl）](https://commoncrawl.org)的大规模网络数据集。通用爬虫是一家非营利组织，负责全网爬取并向公众免费提供其存档与数据集——有关源数据的详细信息，请参阅[其最新爬取公告](https://commoncrawl.org/blog/march-2026-crawl-archive-now-available)。所有页面都将经过处理流程：从WARC记录、HTTP响应头与HTML `<head>` 标签中提取原生HTML主体与结构化元数据，随后将所有内容打包为包含24个字段的Parquet文件。 当前数据集包含爬取批次**CC-MAIN-2026-12**，涵盖**10个分片下的197,357份文档**。我们将34.3 GB的原生HTML处理为34.3 GB的存储主体文本，其中Parquet压缩文件（Zstd压缩）大小为6.5 GB。我们计划后续添加更多爬取快照。 **OpenHTML**采用**开放数据共同体署名许可证（Open Data Commons Attribution License, ODC-By）v1.0**发布，与通用爬虫使用的许可证一致。 ## 本次发布的内容是什么？ 每份通用爬虫的WARC文件（约1 GB压缩HTML）将对应一个Parquet分片。分片存储于与爬取批次对应的目录下，以便多个快照共存： data/ CC-MAIN-2026-12/ 00000.parquet 00001.parquet ... Parquet文件中的每一行代表一个网页，包含**24个元数据字段**。每一行都包含从原始WARC头解析得到的`warc_record_id`与`warc_date`字段，可将任意文档追溯至其源记录。我们还提取了HTTP响应头信息（`content_type`、`charset`、`content_language`、`http_server`、`http_last_modified`）以及HTML `<head>` 元数据（`title`、`description`、`og:title`、`og:description`、`og:image`、`og:type`、`canonical_url`、`html_lang`）。URL会被拆解为`host`（主机名）、`domain`（有效顶级域+1级域，即eTLD+1）、`path`（路径）与`query`（查询字符串）。 ## 如何下载与使用OpenHTML ### 使用`datasets`库 python from datasets import load_dataset # 流式加载完整数据集 ds = load_dataset("open-index/open-html", name="CC-MAIN-2026-12", split="train", streaming=True) for doc in ds: print(doc["url"], doc["title"], len(doc["body"])) # 加载单个分片至内存 ds = load_dataset( "open-index/open-html", data_files="data/CC-MAIN-2026-12/00000.parquet", split="train", ) ### 使用`huggingface_hub`库 python from huggingface_hub import snapshot_download folder = snapshot_download( "open-index/open-html", repo_type="dataset", local_dir="./open-html/", allow_patterns="data/CC-MAIN-2026-12/*", ) 如需加速下载，请安装`pip install huggingface_hub[hf_transfer]`并设置环境变量`HF_HUB_ENABLE_HF_TRANSFER=1`。 ### 使用DuckDB sql SELECT url, title, domain, html_lang, html_length FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') WHERE domain = 'wikipedia.org' LIMIT 10; sql -- 按页面数量排序的顶级域名 SELECT domain, COUNT(*) as pages, AVG(html_length) as avg_html FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') GROUP BY domain ORDER BY pages DESC LIMIT 20; sql -- 包含开放图谱元数据的页面 SELECT url, og_title, og_description, og_image FROM read_parquet('hf://datasets/open-index/open-html/data/CC-MAIN-2026-12/*.parquet') WHERE og_title != '' AND og_image != '' LIMIT 10; # OpenHTML 数据集卡片 ## 数据集描述 - **主页与代码仓库：** [https://huggingface.co/datasets/open-index/open-html](https://huggingface.co/datasets/open-index/open-html) - **联系方式：** 请在社区标签页发起讨论 - **许可证：** 开放数据共同体署名许可证（ODC-By）v1.0 ## 数据集结构 ### 数据实例 以下为数据集中的示例行： json { "url": "https://example.com/article/interesting-topic", "warc_date": "2026-03-05T07:14:58Z", "warc_record_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "warc_filename": "CC-MAIN-20260305070756-20260305100756-00000.warc.gz", "http_status": 200, "content_type": "text/html", "charset": "utf-8", "content_language": "en", "http_server": "nginx", "http_last_modified": "Tue, 04 Mar 2026 12:00:00 GMT", "host": "example.com", "domain": "example.com", "path": "/article/interesting-topic", "query": "", "html_lang": "en", "title": "Interesting Topic - Example", "description": "A fascinating article about interesting topics.", "og_title": "Interesting Topic", "og_description": "A fascinating article about interesting topics.", "og_image": "https://example.com/images/topic.jpg", "og_type": "article", "canonical_url": "https://example.com/article/interesting-topic", "html_length": 48210, "body": "<!DOCTYPE html><html lang="en"><head>..." } ### 数据字段 | 字段名 | 数据类型 | 描述 | |---|---|---| | `url` | 字符串 | 爬取页面的完整URL | | `warc_date` | 字符串 | WARC记录中的爬取时间戳（符合RFC 3339规范） | | `warc_record_id` | 字符串 | 来自WARC-Record-ID头的UUID，用于溯源源记录 | | `warc_filename` | 字符串 | 通用爬虫提供的源WARC文件名 | | `http_status` | int32 | HTTP响应状态码（本数据集内均为200） | | `content_type` | 字符串 | HTTP响应的Content-Type字段（均以`text/html`开头） | | `charset` | 字符串 | Content-Type头中的字符编码（例如`utf-8`、`iso-8859-1`） | | `content_language` | 字符串 | Content-Language HTTP头（例如`en`、`de`、`fr`） | | `http_server` | 字符串 | HTTP响应中的服务器软件（例如`nginx`、`Apache`） | | `http_last_modified` | 字符串 | Last-Modified HTTP头，代表页面最后修改时间 | | `host` | 字符串 | 从URL提取的小写主机名（例如`www.example.com`） | | `domain` | 字符串 | 注册域名（eTLD+1），用于聚合子域名（例如`example.com`） | | `path` | 字符串 | URL路径部分（例如`/article/interesting-topic`） | | `query` | 字符串 | URL查询字符串（若存在，例如`page=2&sort=date`） | | `html_lang` | 字符串 | `<html lang="...">`标签中的语言属性 | | `title` | 字符串 | `<head>`中`<title>`标签对应的页面标题 | | `description` | 字符串 | `<meta name="description">`对应的元描述 | | `og_title` | 字符串 | `<meta property="og:title">`对应的开放图谱标题 | | `og_description` | 字符串 | `<meta property="og:description">`对应的开放图谱描述 | | `og_image` | 字符串 | `<meta property="og:image">`对应的开放图谱图片URL | | `og_type` | 字符串 | `<meta property="og:type">`对应的开放图谱类型（例如`article`、`website`） | | `canonical_url` | 字符串 | `<link rel="canonical">`对应的规范URL，即页面的首选URL | | `html_length` | int64 | 原生HTML主体的字节长度 | | `body` | 字符串 | 原生HTML主体（完整内容，无截断） | ### 数据拆分 默认子集包含所有爬取快照的可用数据。您也可以通过将爬取ID作为配置名称加载特定批次（例如`CC-MAIN-2026-12`）。 ## 数据集创建 ### 策展动机 大多数公开网络数据集要么仅发布原生文本（丢失结构信息），要么发布处理后的Markdown格式（丢失元数据）。**OpenHTML**采用了不同的思路：它保留了**原生HTML**，并附带从WARC头、HTTP响应头与HTML `<head>` 标签中提取的**24个结构化元数据字段**。这使您可以： - 在完整上下文下针对原生网络内容**训练**模型 - 按语言、域名、内容类型或开放图谱元数据进行**筛选** - 分析网络结构、服务器软件分布或字符编码使用情况 - 将每份文档追溯至其精确的WARC源记录 ### 源数据 源数据由[通用爬虫（Common Crawl）](https://commoncrawl.org)基金会爬取的网页组成。通用爬虫归档了全网数十亿个页面，并将原始WARC文件免费发布在Amazon S3上。 ### 数据处理流程 处理流程采用单通道流式提取： 1. **下载**：从通用爬虫的S3存储下载原始.warc.gz文件（每份文件压缩后约1 GB） 2. **筛选**：仅保留HTTP 200响应且Content-Type为`text/html`的页面，丢弃图片、脚本、重定向与错误页面 3. **解析**：提取HTTP响应头中的`content_type`、`charset`、`content_language`、`server`与`last_modified`字段 4. **拆解URL**：将URL拆分为`host`、`domain`（通过公共后缀列表（Public Suffix List）获取eTLD+1）、`path`与`query` 5. **提取元数据**：使用流式分词器提取HTML `<head>` 中的元数据：`title`、`description`、开放图谱标签（`og:title`、`og:description`、`og:image`、`og:type`）、`canonical_url`与`html_lang` 6. **存储**：保存完整的HTML主体（无截断，`html_length`与`body`的实际大小一致） 7. **导出**：直接导出为Apache Parquet格式，采用Zstd压缩，每个行组包含100,000行数据 未生成任何中间文件——流程从压缩的WARC文件流式提取数据，直接转换为Parquet格式。HTML主体为空的页面将被丢弃。 ### 压缩比例 以下数据为CC-MAIN-2026-12批次的10个分片的实际测量值，并投影至100,000个WARC文件的完整爬取规模： | 处理阶段 | 10个分片（实测） | 100,000个分片（投影） | 压缩比例 | |---|---|---|---| | 原始WARC（.warc.gz，已下载） | ~8.1 GB | ~79.2 TB | — | | 提取的HTML（未压缩） | 34.3 GB | ~335.2 TB | — | | 存储的HTML主体 | 34.3 GB | ~335.2 TB | **相较于HTML无压缩** | | 最终Parquet文件（Zstd压缩） | 6.5 GB | ~63.8 TB | **相较于HTML主体压缩81.0%** | `body`字段存储完整的原生HTML。Parquet结合Zstd压缩可进一步压缩数据。整体而言：8.1 GB的原始gzip压缩WARC文件将转换为**6.5 GB的Parquet文件**——总压缩率达**19.5%**，共包含197,357份带有完整元数据的网页。 ### 处理时长 CC-MAIN-2026-12批次的10个分片的处理耗时如下： Download (raw WARC) ████████████████████████ 1h 29m 47s Extract (WARC → HTML + metadata) ███████████████████████░ 1h 28m 15s Publish (HuggingFace上传) ███░░░░░░░░░░░░░░░░░░░░░ 12m 58s ### 数据集图表   ### 个人与敏感信息 除通用爬虫已进行的过滤外，未额外应用任何个人可识别信息（PII）过滤。由于本数据集源自公开网络，可能包含部分个人可识别信息。若您发现自己的PII出现在数据集中并希望移除，请在代码仓库提交Issue。 ## 数据集使用注意事项 ### 社会影响 通过发布本数据集与完整处理流程，我们旨在降低使用高质量网络数据训练与评估大语言模型（Large Language Model, LLM）的门槛。无法承担自行搭建通用爬虫处理流程成本的研究人员与从业者可直接使用**OpenHTML**。 ### 偏差说明 **OpenHTML**继承了通用爬虫与公开网络本身存在的偏差。筛选步骤仅保留`text/html`类型的页面，这可能低估了以其他内容类型提供的内容占比。我们未应用任何基于机器学习的质量或毒性过滤，因为此类过滤器已被证明会不成比例地移除特定方言与社区的内容。 ### 已知局限 本数据集存储完整的HTML主体，未进行截断。超大页面（例如包含内联数据URI的页面）会增加分片大小。`html_length`字段精确反映了HTML主体的字节大小。 为提升性能，元数据提取仅扫描`<head>`部分。若页面将`<meta>`或`<title>`标签放置在`<body>`中，则会丢失对应的元数据。 ## 补充信息 ### 许可证 本数据集采用**开放数据共同体署名许可证（ODC-By）v1.0**发布。使用本数据集同时需遵守[通用爬虫的使用条款](https://commoncrawl.org/terms-of-use)。原始内容仍受其对应发布者的版权与条款约束。 ### 联系方式 如有疑问、反馈或问题，请在[社区标签页](https://huggingface.co/datasets/open-index/open-html/discussions)发起讨论。
