free-law/Caselaw_Access_Project_FAISS_index

--- license: cc0-1.0 task_categories: - text-generation language: - en tags: - legal - law - caselaw pretty_name: Caselaw Access Project size_categories: - 1M<n<10M --- <img src="https://huggingface.co/datasets/TeraflopAI/Caselaw_Access_project/resolve/main/cap.png" width="800"> # The Caselaw Access Project In collaboration with Ravel Law, Harvard Law Library digitized over 40 million U.S. court decisions consisting of 6.7 million cases from the last 360 years into a dataset that is widely accessible to use. Access a bulk download of the data through the Caselaw Access Project API (CAPAPI): https://case.law/caselaw/ Find more information about accessing state and federal written court decisions of common law through the bulk data service documentation here: https://case.law/docs/ Learn more about the Caselaw Access Project and all of the phenomenal work done by Jack Cushman, Greg Leppert, and Matteo Cargnelutti here: https://case.law/about/ Watch a live stream of the data release here: https://lil.law.harvard.edu/about/cap-celebration/stream # Post-processing Teraflop AI is excited to help support the Caselaw Access Project and Harvard Library Innovation Lab, in the release of over 6.6 million state and federal court decisions published throughout U.S. history. It is important to democratize fair access to data to the public, legal community, and researchers. This is a processed and cleaned version of the original CAP data. During the digitization of these texts, there were erroneous OCR errors that occurred. We worked to post-process each of the texts for model training to fix encoding, normalization, repetition, redundancy, parsing, and formatting. Teraflop AI’s data engine allows for the massively parallel processing of web-scale datasets into cleaned text form. Our one-click deployment allowed us to easily split the computation between 1000s of nodes on our managed infrastructure. # FAISS Index We built a FAISS index over all of the post-processed legal texts. The index consists of ~6.6 million dense vectors and the average search speed of a query over the entire index is 12.46 milliseconds. The FAISS library by @Meta allows you to perform k-nearest neighbor search efficiently and in a scalable way over millions of dense vectors. Find the FAISS library here: https://github.com/facebookresearch/faiss The combination of an Inverted File Index (IVF), Product quantization (PQ), and Hierarchical Navigable Small World (HNSW) allows us to run these queries across all of the dense vectors in milliseconds. Find more information about everything here: https://github.com/facebookresearch/faiss/wiki/Faiss-indexes # Licensing Information The Caselaw Access Project dataset is licensed under the [CC0 License](https://creativecommons.org/public-domain/cc0/). # Citation Information ``` The President and Fellows of Harvard University. "Caselaw Access Project." 2024, https://case.law/ ``` ``` @misc{ccap, title={Cleaned Caselaw Access Project}, author={Enrico Shippole, Aran Komatsuzaki}, howpublished{\url{https://huggingface.co/datasets/TeraflopAI/Caselaw_Access_Project}}, year={2024} } ```--- 许可证: CC0 1.0协议 任务类别: - 文本生成 语言: - 英语 标签: - 法律 - 法学 - 判例法 规范名称: 判例法访问项目（Caselaw Access Project） 数据规模分类: - 100万 < 样本数 < 1000万 --- <img src="https://huggingface.co/datasets/TeraflopAI/Caselaw_Access_project/resolve/main/cap.png" width="800"> # 判例法访问项目（Caselaw Access Project） 哈佛大学法学院图书馆与Ravel Law合作，将过去360年间产生的670万件美国法院判决（累计超4000万页文本）数字化为可广泛访问的开放数据集。用户可通过判例法访问项目应用程序编程接口（Caselaw Access Project API, CAPAPI）获取该数据集的批量下载包：https://case.law/caselaw/ 如需了解更多通过批量数据服务文档获取联邦与州级普通法书面法院判决的相关信息，请访问：https://case.law/docs/ 如需了解更多关于本项目以及Jack Cushman、Greg Leppert与Matteo Cargnelutti团队所开展的卓越工作，请访问：https://case.law/about/ 可在此观看数据集发布的直播流：https://lil.law.harvard.edu/about/cap-celebration/stream # 后处理 特弗洛普人工智能（Teraflop AI）很高兴支持判例法访问项目与哈佛大学图书馆创新实验室，发布美国历史上累计的660余万件州级与联邦法院判决。为公众、法律界与研究人员提供公平的数据获取渠道，具有重要的社会价值。本数据集为原始CAP数据的清洗与处理后版本。 在文本数字化过程中，出现了部分光学字符识别（Optical Character Recognition, OCR）错误。我们针对模型训练需求对每份文本进行后处理，以修正编码、标准化、重复冗余、解析与格式问题。 特弗洛普人工智能的数据引擎支持对网页级规模的数据集进行大规模并行处理，将其转换为清洗后的标准文本格式。依托托管计算基础设施，我们的一键部署功能可轻松在数千个计算节点间分配计算任务。 # FAISS索引 我们针对所有经过后处理的法律文本构建了FAISS索引。该索引包含约660万个稠密向量，针对全索引的查询平均搜索速度为12.46毫秒。 Meta开发的FAISS库可针对数百万个稠密向量高效且可扩展地执行k近邻搜索。FAISS库的开源仓库地址为：https://github.com/facebookresearch/faiss 结合倒排文件索引（Inverted File Index, IVF）、乘积量化（Product Quantization, PQ）与分层导航小世界（Hierarchical Navigable Small World, HNSW）算法，我们可在毫秒级时间内完成针对全量稠密向量的查询。更多相关技术细节请访问：https://github.com/facebookresearch/faiss/wiki/Faiss-indexes # 许可信息 本判例法访问项目数据集采用[CC0协议](https://creativecommons.org/public-domain/cc0/)进行许可。 # 引用信息 哈佛大学董事会及院士。"判例法访问项目（Caselaw Access Project）". 2024, https://case.law/ @misc{ccap, title={清洗版判例法访问项目数据集}, author={Enrico Shippole, Aran Komatsuzaki}, howpublished{url{https://huggingface.co/datasets/TeraflopAI/Caselaw_Access_Project}}, year={2024} }