DMOZ 2006 Dataset and its Wikification

This dataset was retrieved with a crawler in 2006 from the Open Directory Project (ODP) (http://dmoz.org, https://en.wikipedia.org/wiki/DMOZ), which closed in 2017 and was reborn as Curlie (https://curlie.org/). The topics were selected from the third level of the ODP hierarchy. Some constraints were imposed on this selection to ensure the quality of the dataset. The minimum size for each selected topic was 100 URLs, and the language was restricted to English. For each topic, we collected all of its URLs as well as those in its subtopics. The retrieved HTML was parsed and cleaned to remove empty, pdf, flash, and other not useful files. The total number of collected pages was more than 350K from 448 topics. In 2018 the data was wikified.

本数据集于2006年通过爬虫从开放目录项目（Open Directory Project，ODP，原站点地址为http://dmoz.org，详细信息可参考https://en.wikipedia.org/wiki/DMOZ）采集获取。该项目于2017年关停，后以Curlie之名重新上线，新站点地址为https://curlie.org/。数据集的主题选自ODP层级结构的第三层级，为保障数据集质量，本次筛选设置了两项约束：其一，所选主题的URL数量下限为100条；其二，仅限定语种为英语。针对每个选定主题，我们采集了该主题及其子主题下的全部URL。对获取到的HTML内容进行了解析与清洗，剔除了空文件、PDF文件、Flash文件及其他无实用价值的文件。本次采集共从448个主题中获得超过35万个网页页面。该数据集于2018年完成了维基化处理。

搜集汇总

背景与挑战

背景概述

该数据集基于2006年从开放目录项目（ODP）爬取的数据，专注于英语内容，主题选自ODP层次结构的第三级，并设定了最小URL数量等质量约束以确保数据可靠性。数据集经过解析和清理，移除了无用文件，包含超过35万页，涉及448个主题，并在2018年进行了wikification处理，以增强其结构化和可链接性。