BeIR/quora

--- language: - en license: - cc-by-sa-4.0 multilinguality: - monolingual paperswithcode_id: beir pretty_name: BEIR Benchmark task_categories: - zero-shot-classification - text-retrieval task_ids: - document-retrieval - entity-linking-retrieval - fact-checking-retrieval tags: - biomedical-information-retrieval - citation-prediction-retrieval - passage-retrieval - news-retrieval - argument-retrieval - zero-shot-information-retrieval - tweet-retrieval - question-answering-retrieval - duplication-question-retrieval - zero-shot-retrieval configs: - config_name: corpus data_files: - split: corpus path: corpus/corpus-* - config_name: queries data_files: - split: queries path: queries/queries-* dataset_info: - config_name: corpus features: - name: _id dtype: string - name: title dtype: string - name: text dtype: string splits: - name: corpus num_bytes: 23601001 num_examples: 522931 download_size: 23601001 dataset_size: 23601001 - config_name: queries features: - name: _id dtype: string - name: title dtype: string - name: text dtype: string splits: - name: queries num_bytes: 647974 num_examples: 15000 download_size: 647974 dataset_size: 647974 --- # Dataset Card for BEIR Benchmark > **`quora` is one of the datasets from the Duplicate Question Retrieval task within BEIR, measuring duplicate query retrieval for a given query.** > **NOTE: ArguAna has queries also incorporated within the corpus, so you should remove the same query_id if present within the corpus during inference (implemented in BEIR)** ## Dataset Description - **Homepage:** https://beir.ai - **Repository:** https://beir.ai - **Paper:** https://openreview.net/forum?id=wCu6T5xFjeJ - **Leaderboard:** https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns - **Point of Contact:** [e-mail] ### Dataset Summary BEIR is a heterogeneous benchmark built from 18 diverse datasets representing 9 information retrieval tasks. - Fact-checking: [FEVER](http://fever.ai), [Climate-FEVER](http://climatefever.ai), [SciFact](https://github.com/allenai/scifact) - Question-Answering: [NQ](https://ai.google.com/research/NaturalQuestions), [HotpotQA](https://hotpotqa.github.io), [FiQA-2018](https://sites.google.com/view/fiqa/) - Bio-Medical IR: [TREC-COVID](https://ir.nist.gov/covidSubmit/index.html), [BioASQ](http://bioasq.org), [NFCorpus](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) - News Retrieval: [TREC-NEWS](https://trec.nist.gov/data/news2019.html), [Robust04](https://trec.nist.gov/data/robust/04.guidelines.html) - Argument Retrieval: [Touche-2020](https://webis.de/events/touche-20/shared-task-1.html), [ArguAna](tp://argumentation.bplaced.net/arguana/data) - Duplicate Question Retrieval: [Quora](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs), [CqaDupstack](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/) - Citation-Prediction: [SCIDOCS](https://allenai.org/data/scidocs) - Tweet Retrieval: [Signal-1M](https://research.signal-ai.com/datasets/signal1m-tweetir.html) - Entity Retrieval: [DBPedia](https://github.com/iai-group/DBpedia-Entity/) ### Languages All tasks are in English (`en`). ## Dataset Structure This dataset uses the standard BEIR retrieval layout and includes: - `corpus`: one row per document with `_id`, `title`, `text` - `queries`: one row per query with `_id`, `title`, `text` ### Data Fields - `_id` (`string`): unique identifier - `title` (`string`): title (empty string when unavailable) - `text` (`string`): document/query text ### Data Instances A high level example of any BEIR dataset: ```python corpus = { "doc1" : { "title": "Albert Einstein", "text": "Albert Einstein was a German-born theoretical physicist. who developed the theory of relativity, \ one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for \ its influence on the philosophy of science. He is best known to the general public for his mass–energy \ equivalence formula E = mc2, which has been dubbed 'the world's most famous equation'. He received the 1921 \ Nobel Prize in Physics 'for his services to theoretical physics, and especially for his discovery of the law \ of the photoelectric effect', a pivotal step in the development of quantum theory." }, "doc2" : { "title": "", # Keep title an empty string if not present "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat relative to the amount of \ malted barley. The two main varieties are German Weißbier and Belgian witbier; other types include Lambic (made\ with wild yeast), Berliner Weisse (a cloudy, sour beer), and Gose (a sour, salty beer)." }, } queries = { "q1" : "Who developed the mass-energy equivalence formula?", "q2" : "Which beer is brewed with a large proportion of wheat?" } qrels = { "q1" : {"doc1": 1}, "q2" : {"doc2": 1}, } ``` ### Quora Data Splits | Subset | Split | Rows | | --- | --- | ---: | | corpus | corpus | 522,931 | | queries | queries | 15,000 | ### BEIR Direct Download You can also download BEIR datasets directly (without loading through Hugging Face datasets) using the links below. | Dataset | Website | BEIR-Name | Type | Queries | Corpus | Rel D/Q | Down-load | md5 | | --- | --- | --- | --- | ---: | ---: | ---: | --- | --- | | MSMARCO | [Homepage](https://microsoft.github.io/msmarco/) | `msmarco` | `train` `dev` `test` | 6,980 | 8.84M | 1.1 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/msmarco.zip) | `444067daf65d982533ea17ebd59501e4` | | TREC-COVID | [Homepage](https://ir.nist.gov/covidSubmit/index.html) | `trec-covid` | `test` | 50 | 171K | 493.5 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid.zip) | `ce62140cb23feb9becf6270d0d1fe6d1` | | NFCorpus | [Homepage](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) | `nfcorpus` | `train` `dev` `test` | 323 | 3.6K | 38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nfcorpus.zip) | `a89dba18a62ef92f7d323ec890a0d38d` | | BioASQ | [Homepage](http://bioasq.org) | `bioasq` | `train` `test` | 500 | 14.91M | 8.05 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#2-bioasq) | | NQ | [Homepage](https://ai.google.com/research/NaturalQuestions) | `nq` | `train` `test` | 3,452 | 2.68M | 1.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq.zip) | `d4d3d2e48787a744b6f6e691ff534307` | | HotpotQA | [Homepage](https://hotpotqa.github.io) | `hotpotqa` | `train` `dev` `test` | 7,405 | 5.23M | 2.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/hotpotqa.zip) | `f412724f78b0d91183a0e86805e16114` | | FiQA-2018 | [Homepage](https://sites.google.com/view/fiqa/) | `fiqa` | `train` `dev` `test` | 648 | 57K | 2.6 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip) | `17918ed23cd04fb15047f73e6c3bd9d9` | | Signal-1M(RT) | [Homepage](https://research.signal-ai.com/datasets/signal1m-tweetir.html) | `signal1m` | `test` | 97 | 2.86M | 19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#4-signal-1m) | | TREC-NEWS | [Homepage](https://trec.nist.gov/data/news2019.html) | `trec-news` | `test` | 57 | 595K | 19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#1-trec-news) | | ArguAna | [Homepage](http://argumentation.bplaced.net/arguana/data) | `arguana` | `test` | 1,406 | 8.67K | 1.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/arguana.zip) | `8ad3e3c2a5867cdced806d6503f29b99` | | Touche-2020 | [Homepage](https://webis.de/events/touche-20/shared-task-1.html) | `webis-touche2020` | `test` | 49 | 382K | 19.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/webis-touche2020.zip) | `46f650ba5a527fc69e0a6521c5a23563` | | CQADupstack | [Homepage](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/) | `cqadupstack` | `test` | 13,145 | 457K | 1.4 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/cqadupstack.zip) | `4e41456d7df8ee7760a7f866133bda78` | | Quora | [Homepage](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) | `quora` | `dev` `test` | 10,000 | 523K | 1.6 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/quora.zip) | `18fb154900ba42a600f84b839c173167` | | DBPedia | [Homepage](https://github.com/iai-group/DBpedia-Entity/) | `dbpedia-entity` | `dev` `test` | 400 | 4.63M | 38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/dbpedia-entity.zip) | `c2a39eb420a3164af735795df012ac2c` | | SCIDOCS | [Homepage](https://allenai.org/data/scidocs) | `scidocs` | `test` | 1,000 | 25K | 4.9 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scidocs.zip) | `38121350fc3a4d2f48850f6aff52e4a9` | | FEVER | [Homepage](http://fever.ai) | `fever` | `train` `dev` `test` | 6,666 | 5.42M | 1.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fever.zip) | `5a818580227bfb4b35bb6fa46d9b6c03` | | Climate-FEVER | [Homepage](http://climatefever.ai) | `climate-fever` | `test` | 1,535 | 5.42M | 3.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/climate-fever.zip) | `8b66f0a9126c521bae2bde127b4dc99d` | | SciFact | [Homepage](https://github.com/allenai/scifact) | `scifact` | `train` `test` | 300 | 5K | 1.1 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip) | `5f7d1de60b170fc8027bb7898e2efca1` | | Robust04 | [Homepage](https://trec.nist.gov/data/robust/04.guidelines.html) | `robust04` | `test` | 249 | 528K | 69.9 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#3-robust04) | ## Citation Information ```bibtex @inproceedings{ thakur2021beir, title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models}, author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych}, booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)}, year={2021}, url={https://openreview.net/forum?id=wCu6T5xFjeJ} } ```

原始信息汇总

数据集卡片 for BEIR Benchmark

数据集描述

数据集摘要

BEIR 是一个异构基准，由 18 个不同数据集组成，代表 9 种信息检索任务：

• 事实检查：FEVER, Climate-FEVER, SciFact
• 问答：NQ, HotpotQA, FiQA-2018
• 生物医学信息检索：TREC-COVID, BioASQ, NFCorpus
• 新闻检索：TREC-NEWS, Robust04
• 论点检索：Touche-2020, ArguAna
• 重复问题检索：Quora, CqaDupstack
• 引文预测：SCIDOCS
• 推文检索：Signal-1M
• 实体检索：DBPedia

所有这些数据集都已预处理，可供实验使用。

支持的任务和排行榜

数据集支持排行榜，评估模型在任务特定指标（如 F1 或 EM）以及从 Wikipedia 检索支持信息的能力。

语言

所有任务均为英语 (en)。

数据集结构

所有 BEIR 数据集必须包含语料库、查询和 qrels（相关性判断文件）。它们必须采用以下格式：

• corpus 文件：一个 .jsonl 文件（jsonlines），包含一个字典列表，每个字典有三个字段 _id（唯一文档标识符）、title（文档标题，可选）和 text（文档段落或段落）。
• queries 文件：一个 .jsonl 文件（jsonlines），包含一个字典列表，每个字典有两个字段 _id（唯一查询标识符）和 text（查询文本）。
• qrels 文件：一个 .tsv 文件（制表符分隔），包含三列，即 query-id、corpus-id 和 score（查询和文档的相关性判断）。

数据实例

一个高层次的 BEIR 数据集示例：

python corpus = { "doc1": { "title": "Albert Einstein", "text": "Albert Einstein was a German-born theoretical physicist..." }, "doc2": { "title": "", "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat..." }, }

queries = { "q1": "Who developed the mass-energy equivalence formula?", "q2": "Which beer is brewed with a large proportion of wheat?" }

qrels = { "q1": {"doc1": 1}, "q2": {"doc2": 1}, }

数据字段

所有配置的示例具有以下特征：

语料库

• corpus：一个 dict 特征，表示文档标题和段落文本，由以下部分组成：
  • _id：一个 string 特征，表示唯一文档 ID。
  • title：一个 string 特征，表示文档标题。
  • text：一个 string 特征，表示文档文本。

查询

• queries：一个 dict 特征，表示查询，由以下部分组成：
  • _id：一个 string 特征，表示唯一查询 ID。
  • text：一个 string 特征，表示查询文本。

Qrels

• qrels：一个 dict 特征，表示查询文档相关性判断，由以下部分组成：
  • _id：一个 string 特征，表示查询 ID。
  • _id：一个 string 特征，表示文档 ID。
  • score：一个 int32 特征，表示查询和文档的相关性判断。

数据分割

数据集	网站	BEIR 名称	类型	查询数量	语料库大小	相关文档/查询	下载链接	md5
MSMARCO	Homepage	msmarco	train<br>dev<br>test	6,980	8.84M	1.1	Link	444067daf65d982533ea17ebd59501e4
TREC-COVID	Homepage	trec-covid	test	50	171K	493.5	Link	ce62140cb23feb9becf6270d0d1fe6d1
NFCorpus	Homepage	nfcorpus	train<br>dev<br>test	323	3.6K	38.2	Link	a89dba18a62ef92f7d323ec890a0d38d
BioASQ	Homepage	bioasq	train<br>test	500	14.91M	8.05	No	How to Reproduce?
NQ	Homepage	nq	train<br>test	3,452	2.68M	1.2	Link	d4d3d2e48787a744b6f6e691ff534307
HotpotQA	Homepage	hotpotqa	train<br>dev<br>test	7,405	5.23M	2.0	Link	f412724f78b0d91183a0e86805e16114
FiQA-2018	Homepage	fiqa	train<br>dev<br>test	648	57K	2.6	Link	17918ed23cd04fb15047f73e6c3bd9d9
Signal-1M(RT)	Homepage	signal1m	test	97	2.86M	19.6	No	How to Reproduce?
TREC-NEWS	Homepage	trec-news	test	57	595K	19.6	No	How to Reproduce?
ArguAna	Homepage	arguana	test	1,406	8.67K	1.0	Link	8ad3e3c2a5867cdced806d6503f29b99
Touche-2020	Homepage	webis-touche2020	test	49	382K	19.0	Link	46f660ba5a527fc69e0a6521c5a23563
CQADupstack	Homepage	cqadupstack	test	13,145	457K	1.4	Link	4e41456d7df8ee7760a7f866133bda78
Quora	Homepage	quora	dev<br>test	10,000	523K	1.6	Link	18fb154900ba42a600f84b839c173167
DBPedia	Homepage	dbpedia-entity	dev<br>test	400	4.63M	38.2	Link	c2a39eb420a3164af735795df012ac2c
SCIDOCS	Homepage	scidocs	test	1,000	25K	4.9	Link	38121350fc3a4d2f48850f6aff52e4a9
FEVER	Homepage	fever	train<br>dev<br>test	6,666	5.42M	1.2	Link	5a818580227bfb4b35bb6fa46d9b6c03
Climate-FEVER	Homepage	climate-fever	test	1,535	5.42M	3.0	Link	8b66f0a9126c521bae2bde127b4dc99d
SciFact	Homepage	scifact	train<br>test	300	5K	1.1	Link	5f7d1de60b170fc8027bb7898e2efca1
Robust04	Homepage	robust04	test	249	528K	69.9	No	How to Reproduce?

数据集创建

策划理由

[需要更多信息]

源数据

初始数据收集和规范化

[需要更多信息]

源语言生产者是谁？

[需要更多信息]

注释

注释过程

[需要更多信息]

注释者是谁？

[需要更多信息]

个人和敏感信息

[需要更多信息]

使用数据的注意事项

数据集的社会影响

[需要更多信息]

偏见的讨论

[需要更多信息]

其他已知限制

[需要更多信息]

附加信息

数据集策展人

[需要更多信息]

许可信息

[需要更多信息]

引用信息

引用为：

@inproceedings{ thakur2021beir, title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models}, author={Nandan Thakur and Nils Reimers and Andreas R{"u}ckl{e} and Abhishek Srivastava and Iryna Gurevych}, booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)}, year={2021}, url={https://openreview.net/forum?id=wCu6T5xFjeJ} }

贡献

感谢 @Nthakur20 添加此数据集。

搜集汇总

背景与挑战

背景概述

BeIR/quora是BEIR基准测试中的数据集，专注于重复问题检索任务，用于评估信息检索模型的零样本性能。数据集包含超过52万行的语料库和1.5万行的查询数据，均为英文文本，适用于文档检索和实体链接等子任务。