面向互联网信息服务内容安全的百万级多类别内容资源库数据

面向互联网信息服务内容安全治理的高精度需求，本项目采用系统化的数据工程方法构建了面向互联网信息服务内容安全的百万级多类别内容资源库数据。数据集以2020—2024年为时间轴，融合了Weibo-COV开源数据、清华大学COLDataset以及定向采集的今日头条热点舆情，构建了覆盖多源场景的综合样本体系。本资源库包含文本记录总数超514万条。在构建过程中，利用自动化采集脚本获取元数据，采用SimHash高效哈希算法结合编辑距离进行语义去重，并严格对标《网络信息内容生态治理规定》，通过专家复核与人工抽检相结合的闭环管理机制，对“正能量、中立、有害”三分类标签进行复核。数据集以结构化CSV格式存储，不仅填补了中文社交媒体在精细化安全语义识别领域的空白，更为个性化推荐算法的安全性评测、风险过滤机制优化及计算社会学研究提供了可复现、高置信度的标准化支撑。To meet the high-precision requirements of content security governance for Internet information services, this project adopts a systematic data engineering approach to construct a million-scale multi-category content resource dataset for Internet information service content security. Spanning the period from 2020 to 2024, the dataset integrates the Weibo-COV open-source dataset, COLDataset from Tsinghua University, and targeted-collected hot public opinions from Jinri Toutiao, building a comprehensive sample system covering multi-source scenarios. This resource library contains over 5.14 million text records in total. During the construction process, automated collection scripts are used to obtain metadata, the SimHash efficient hashing algorithm combined with edit distance is applied for semantic deduplication, and strict alignment is conducted with the Regulations on the Governance of Online Information Content Ecosystem. A closed-loop management mechanism combining expert review and manual spot checks is adopted to verify the three-category labels of "positive", "neutral", and "harmful". Stored in structured CSV format, this dataset not only fills the gap in fine-grained security semantic recognition for Chinese social media, but also provides reproducible, high-confidence standardized support for security evaluation of personalized recommendation algorithms, optimization of risk filtering mechanisms, and computational sociology research.

搜集汇总

背景与挑战

背景概述

该数据集为面向互联网信息服务内容安全治理而构建的百万级多类别内容资源库，涵盖2020年至2024年期间来自微博、清华大学及今日头条等多源的超过514万条文本数据。通过SimHash算法和专家复核进行去重与分类标注，以结构化CSV格式存储，旨在支持精细化安全语义识别、个性化推荐算法安全性评测等研究与应用。