baridhi/SEC-EDGAR

SEC-EDGAR数据集是由Datamule、Teraflop AI和Eventual合作发布的金融领域数据集，基于美国证券交易委员会（SEC）的EDGAR数据库。数据集包含590GB的数据，涵盖约800万个样本和430亿个标记，涉及所有主要文件类型，如10-K、8-K、Form 4等。数据通过datamule-python库和官方API收集，原始内容包括每个文件的原始内容、从HTML/XML解析的纯文本以及元数据（如访问号、提交日期、公司信息等）。处理过程中使用了selectolax HTML解析器和修改版的doc2dict、secsgml库来提取文本和处理SGML格式。元数据涵盖文件类型、序列、公司名称、中央索引键等。数据集适用于大型语言模型训练和检索管道构建，主要用于文本生成和分类任务。

The SEC-EDGAR dataset is a collaborative release by Datamule, Teraflop AI, and Eventual, based on the SEC EDGAR database. It contains 590 GB of data, spanning 8 million samples and 43 billion tokens from all major filing types, such as 10-K, 8-K, and Form 4. Data was collected using the datamule-python library and official API, with raw contents including original filings, HTML/XML parsed plaintext, and metadata like accession numbers, filing dates, and company details. Processing involved the selectolax HTML parser and modified versions of doc2dict and secsgml libraries for text extraction and SGML handling. Metadata covers file types, sequences, company names, Central Index Keys, and more. The dataset is designed for training large language models and building retrieval pipelines, primarily for text-generation and text-classification tasks.