SciRIFF

# SciRIFF

The SciRIFF dataset includes 137K instruction-following demonstrations for 54 scientific literature understanding tasks. The tasks cover five essential scientific literature categories and span five domains. The dataset is described in our paper [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://arxiv.org/abs/2406.07835).

There are three dataset configurations with different max context lengths: 4096, 8192, and 16384. All experiments in the paper are performed with the 4096 context window. You can load the dataset like:

```python
import datasets
ds = datasets.load_dataset("allenai/SciRIFF", "4096")
```

Code to create the dataset, train models on SciRIFF, and perform evaluation is available at our GitHub repo: https://github.com/allenai/SciRIFF. To train models on SciRIFF data, you should use the [SciRIFF train mix](https://huggingface.co/datasets/allenai/SciRIFF-train-mix) dataset.

**Table of Contents**

- [Dataset details](#dataset-details)
- [License](#license)
- [Task provenance](#task-provenance)
- [Task metadata](#task-metadata)

## Dataset details

Each instance in SciRIFF has the following fields:

- `input`: Task input (i.e. user message).
- `output`: Task output (i.e. expected model response).
- `_instance_id`: A unique id for the instance, formatted like `{task_name}:{split}:{instance_id}`. For instance, `qasa_abstractive_qa:test:182`.
- `metadata`: Task metadata. More information on the schema for task metadata can be found in the [SciRIFF GitHub repo](https://github.com/allenai/SciRIFF).
- `task_family`: The category to which this task belongs. Options include `summarization`, `ie`, `qa`, `entailment`, and `classification`. Some categories have sub-categories which are largely self-explanatory; see the [repo](https://github.com/allenai/SciRIFF) for more information.
- `domains`: Scientific field(s) that the task covers. Options include: `clinical_medicine`, `biomedicine`, `chemistry`, `artificial_intelligence`, `materials_science`, and `misc`.
- `input_context`: Whether the input is a paragraph, full text, etc. Options include: `sentence`, `paragraph`, `multiple_paragraphs` (including full paper text), and `structured` (e.g. code for a LaTex table).
- `source_type`: Indicates whether the input comes from a single paper or multiple. Options include `single_source`, `multiple_source`.
- `output_context`: Options include: `label`, `sentence`, `paragraph`, `multiple_paragraphs`, `json`, `jsonlines`.

## License

SciRIFF is licensed under `ODC-By`. Licenses of the datasets from which SciRIFF is derived are listed [below](#task-provenance).

## Task provenance

SciRIFF was created by repurposing existing scientific literature understanding datasets. Below we provide information on the source data for each SciRIFF task, including license information on individual datasets where available. Where possible, we leveraged the [BigBIO](https://github.com/bigscience-workshop/biomedical) collection as a starting point, rather than reprocessing datasets from scratch. In the table below, we include the name of the BigBio subset for all tasks available in BigBio; these can be loaded like `datasets.load_dataset(bigbio/{bigbio_subset})`.

| SciRIFF Name | Paper Link | License | Website / Download Link | BigBio Subset |
| :---------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------- | :----------------------------------------------------------------------------------------- | :----------------- |
| `acl_arc_intent_classification` | [ACL ARC](https://aclanthology.org/L08-1005/) | - | <https://github.com/allenai/scicite/> | |
| `anat_em_ner` | [AnatEM](https://academic.oup.com/bioinformatics/article/30/6/868/285282) | CC BY | <https://nactem.ac.uk/anatomytagger/#AnatEM> | `anat_em` |
| `annotated_materials_syntheses_events` | [Materials Science Procedural Text Corpus](https://aclanthology.org/W19-4007/) | MIT | <https://github.com/olivettigroup/annotated-materials-syntheses> | |
| `bc7_litcovid_topic_classification` | [BioCreative VII LitCOVID](https://pubmed.ncbi.nlm.nih.gov/36043400/) | - | <https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vii/track-5/> | `bc7_litcovid` |
| `bioasq_{factoid,general,list,yesno}_qa` | [BioASQ](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0564-6) | CC BY | <http://bioasq.org/> | `bioasq` |
| `biored_ner` | [BioRED](https://academic.oup.com/bib/article/23/5/bbac282/6645993) | - | <https://ftp.ncbi.nlm.nih.gov/pub/lu/BioRED/> | `biored` |
| `cdr_ner` | [BioCreative V CDR](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4860626/) | - | <https://biocreative.bioinformatics.udel.edu/tasks/biocreative-v/track-3-cdr/> | `bc5cdr` |
| `chemdner_ner` | [CHEMDNER](https://jcheminf.biomedcentral.com/articles/10.1186/1758-2946-7-S1-S2) | - | <https://biocreative.bioinformatics.udel.edu/resources/biocreative-iv/chemdner-corpus/> | `chemdner` |
| `chemprot_{ner,re}` | [BioCreative VI ChemProt](https://www.semanticscholar.org/paper/Overview-of-the-BioCreative-VI-chemical-protein-Krallinger-Rabal/eed781f498b563df5a9e8a241c67d63dd1d92ad5) | - | <https://biocreative.bioinformatics.udel.edu/news/corpora/chemprot-corpus-biocreative-vi/> | `chemprot` |
| `chemsum_single_document_summarization` | [ChemSum](https://aclanthology.org/2023.acl-long.587/) | - | <https://github.com/griff4692/calibrating-summaries> | |
| `chemtables_te` | [ChemTables](https://arxiv.org/abs/2305.14336) | GPL 3.0 | <https://huggingface.co/datasets/fbaigt/schema-to-json> | |
| `chia_ner` | [Chia](https://www.nature.com/articles/s41597-020-00620-0) | CC BY | <https://github.com/WengLab-InformaticsResearch/CHIA> | `chia` |
| `covid_deepset_qa` | [COVID-QA](https://aclanthology.org/2020.nlpcovid19-acl.18/) | Apache 2.0 | <https://github.com/deepset-ai/COVID-QA> | `covid_qa_deepset` |
| `covidfact_entailment` | [CovidFact](https://aclanthology.org/2021.acl-long.165/) | - | <https://github.com/asaakyan/covidfact> | |
| `craftchem_ner` | [CRAFT-Chem](https://link.springer.com/chapter/10.1007/978-94-024-0881-2_53) | - | <https://huggingface.co/datasets/ghadeermobasher/CRAFT-Chem> | |
| `data_reco_mcq_{mc,sc}` | [DataFinder](https://aclanthology.org/2023.acl-long.573/) | Apache 2.0 | <https://github.com/viswavi/datafinder/tree/main> | |
| `ddi_ner` | [DDI](https://www.sciencedirect.com/science/article/pii/S1532046413001123) | CC BY | <https://github.com/isegura/DDICorpus> | `ddi_corpus` |
| `discomat_te` | [DISCoMaT](https://aclanthology.org/2023.acl-long.753/) | CC BY-SA | <https://github.com/M3RG-IITD/DiSCoMaT> | |
| `drug_combo_extraction_re` | [Drug Combinations](https://aclanthology.org/2022.naacl-main.233/) | - | <https://github.com/allenai/drug-combo-extraction> | |
| `evidence_inference` | [Evidence inference](https://aclanthology.org/2020.bionlp-1.13/) | MIT | <https://evidence-inference.ebm-nlp.com/> | |
| `genia_ner` | [JNLPBA](https://aclanthology.org/W04-1213/) | CC BY | <https://github.com/spyysalo/jnlpba> | `jnlpba` |
| `gnormplus_ner` | [GNormPlus](https://www.hindawi.com/journals/bmri/2015/918710/) | - | <https://www.ncbi.nlm.nih.gov/research/bionlp/Tools/gnormplus/> | `gnormplus` |
| `healthver_entailment` | [HealthVer](https://aclanthology.org/2021.findings-emnlp.297/) | nan | <https://github.com/sarrouti/healthver> | |
| `linnaeus_ner` | [LINNAEUS](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-85) | CC BY | <https://sourceforge.net/projects/linnaeus/> | `linnaeus` |
| `medmentions_ner` | [MedMentions](https://arxiv.org/abs/1902.09476) | CC 0 | <https://github.com/chanzuckerberg/MedMentions> | `medmentions` |
| `mltables_te` | [AxCell](https://aclanthology.org/2020.emnlp-main.692/) | Apache 2.0 | <https://github.com/paperswithcode/axcell> | |
| `mslr2022_cochrane_multidoc_summarization` | [Cochrane](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8378607/) | Apache 2.0 | <https://github.com/allenai/mslr-shared-task> | |
| `mslr2022_ms2_multidoc_summarization` | [MS^2](https://aclanthology.org/2021.emnlp-main.594/) | Apache 2.0 | <https://github.com/allenai/mslr-shared-task> | |
| `multicite_intent_classification` | [MultiCite](https://aclanthology.org/2022.naacl-main.137/) | CC BY-NC | <https://github.com/allenai/multicite> | |
| `multixscience_multidoc_summarization` | [Multi-XScience](https://aclanthology.org/2020.emnlp-main.648/) | MIT | <https://github.com/yaolu/Multi-XScience> | |
| `mup_single_document_summarization` | [MUP](https://aclanthology.org/2022.sdp-1.32/) | Apache 2.0 | <https://github.com/allenai/mup> | |
| `ncbi_ner` | [NCBI Disease](https://pubmed.ncbi.nlm.nih.gov/24393765/) | CC 0 | <https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/> | `ncbi_disease` |
| `nlmchem_ner` | [NLM-Chem](https://pubmed.ncbi.nlm.nih.gov/33767203/) | CC 0 | <https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/> | `nlmchem` |
| `nlmgene_ner` | [NLM-Gene](https://pubmed.ncbi.nlm.nih.gov/33839304/) | CC 0 | <https://ftp.ncbi.nlm.nih.gov/pub/lu/NLMGene/> | `nlm_gene` |
| `pico_ner` | [EBM-NLP PICO](https://aclanthology.org/P18-1019/) | - | <https://github.com/bepnye/EBM-NLP> | `pico_extraction` |
| `pubmedqa_qa` | [PubMedQA](https://aclanthology.org/D19-1259/) | MIT | <https://github.com/pubmedqa/pubmedqa> | `pubmed_qa` |
| `qasa_abstractive_qa` | [QASA](https://proceedings.mlr.press/v202/lee23n) | MIT | <https://github.com/lgresearch/QASA> | |
| `qasper_{abstractive,extractive}_qa` | [Qasper](https://aclanthology.org/2021.naacl-main.365/) | CC BY | <https://allenai.org/data/qasper> | |
| `scicite_classification` | [SciCite](https://aclanthology.org/N19-1361/) | - | <https://allenai.org/data/scicite> | |
| `scientific_lay_summarisation_`<br>`{elife,plos}_single_doc_summ` | [Lay Summarisation](https://aclanthology.org/2022.emnlp-main.724/) | - | <https://github.com/TGoldsack1/Corpora_for_Lay_Summarisation> | |
| `scientific_papers_summarization_`<br>`single_doc_{arxiv,pubmed}` | [Scientific Papers](https://aclanthology.org/N18-2097/) | - | <https://huggingface.co/datasets/armanc/scientific_papers> | |
| `scierc_{ner,re}` | [SciERC](https://aclanthology.org/D18-1360/) | - | <http://nlp.cs.washington.edu/sciIE/> | |
| `scifact_entailment` | [SciFact](https://aclanthology.org/2020.emnlp-main.609/) | CC BY-NC | <https://allenai.org/data/scifact> | |
| `scireviewgen_multidoc_summarization` | [SciReviewGen](https://aclanthology.org/2023.findings-acl.418/) | CC BY-NC | <https://github.com/tetsu9923/SciReviewGen> | |
| `scitldr_aic` | [SciTLDR](https://aclanthology.org/2020.findings-emnlp.428/) | Apache 2.0 | <https://github.com/allenai/scitldr> | |

## Task metadata

Below we include metadata on each task, as described in the metadata fields [above](#dataset-details).

| SciRIFF Name | Task Family | Domains | Input Context | Source Type | Output Context |
| :--------------------------------------------------------- | :-------------------------- | :----------------------------------------------------------------- | :------------------ | :-------------- | :------------- |
| `acl_arc_intent_classification` | classification | artificial_intelligence | multiple_paragraphs | single_source | label |
| `anat_em_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `annotated_materials_syntheses_events` | ie.event_extraction | materials_science | paragraph | single_source | json |
| `bc7_litcovid_topic_classification` | classification | clinical_medicine | paragraph | single_source | json |
| `bioasq_factoid_qa` | qa.abstractive | biomedicine | multiple_paragraphs | multiple_source | sentence |
| `bioasq_general_qa` | qa.abstractive | biomedicine | multiple_paragraphs | multiple_source | sentence |
| `bioasq_list_qa` | qa.abstractive | biomedicine | multiple_paragraphs | multiple_source | json |
| `bioasq_yesno_qa` | qa.yes_no | biomedicine | multiple_paragraphs | multiple_source | label |
| `biored_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `cdr_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `chemdner_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `chemprot_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `chemprot_re` | ie.relation_extraction | biomedicine | paragraph | single_source | json |
| `chemsum_single_document_summarization` | summarization | chemistry | multiple_paragraphs | single_source | paragraph |
| `chemtables_te` | ie.structure_to_json | chemistry | structured | single_source | jsonlines |
| `chia_ner` | ie.named_entity_recognition | clinical_medicine | paragraph | single_source | json |
| `covid_deepset_qa` | qa.extractive | biomedicine | paragraph | single_source | sentence |
| `covidfact_entailment` | entailment | biomedicine, clinical_medicine | paragraph | single_source | json |
| `craftchem_ner` | ie.named_entity_recognition | biomedicine | sentence | single_source | json |
| `data_reco_mcq_mc` | qa.multiple_choice | artificial_intelligence | multiple_paragraphs | multiple_source | json |
| `data_reco_mcq_sc` | qa.multiple_choice | artificial_intelligence | multiple_paragraphs | multiple_source | label |
| `ddi_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `discomat_te` | ie.structure_to_json | materials_science | structured | single_source | jsonlines |
| `drug_combo_extraction_re` | ie.relation_extraction | clinical_medicine | paragraph | single_source | json |
| `evidence_inference` | ie.relation_extraction | clinical_medicine | paragraph | single_source | json |
| `genia_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `gnormplus_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `healthver_entailment` | entailment | clinical_medicine | paragraph | single_source | json |
| `linnaeus_ner` | ie.named_entity_recognition | biomedicine | multiple_paragraphs | single_source | json |
| `medmentions_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `mltables_te` | ie.structure_to_json | artificial_intelligence | structured | single_source | jsonlines |
| `mslr2022_cochrane_multidoc_summarization` | summarization | clinical_medicine | paragraph | multiple_source | paragraph |
| `mslr2022_ms2_multidoc_summarization` | summarization | clinical_medicine | paragraph | multiple_source | paragraph |
| `multicite_intent_classification` | classification | artificial_intelligence | paragraph | single_source | json |
| `multixscience_multidoc_summarization` | summarization | artificial_intelligence, biomedicine, <br> materials_science, misc | multiple_paragraphs | multiple_source | paragraph |
| `mup_single_document_summarization` | summarization | artificial_intelligence | multiple_paragraphs | single_source | paragraph |
| `ncbi_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `nlmchem_ner` | ie.named_entity_recognition | biomedicine | multiple_paragraphs | single_source | json |
| `nlmgene_ner` | ie.named_entity_recognition | biomedicine | paragraph | single_source | json |
| `pico_ner` | ie.named_entity_recognition | clinical_medicine | paragraph | single_source | json |
| `pubmedqa_qa` | qa.yes_no | biomedicine | paragraph | single_source | label |
| `qasa_abstractive_qa` | qa.abstractive | artificial_intelligence | multiple_paragraphs | single_source | paragraph |
| `qasper_abstractive_qa` | qa.abstractive | artificial_intelligence | multiple_paragraphs | single_source | json |
| `qasper_extractive_qa` | qa.extractive | artificial_intelligence | multiple_paragraphs | single_source | json |
| `scicite_classification` | classification | artificial_intelligence | paragraph | single_source | label |
| `scientific_lay_summarisation_`<br>`elife_single_doc_summ` | summarization | biomedicine | multiple_paragraphs | single_source | paragraph |
| `scientific_lay_summarisation_`<br>`plos_single_doc_summ` | summarization | biomedicine | multiple_paragraphs | single_source | paragraph |
| `scientific_papers_summarization_single_doc_arxiv` | summarization | artificial_intelligence, misc | multiple_paragraphs | single_source | paragraph |
| `scientific_papers_summarization_single_doc_pubmed` | summarization | biomedicine | multiple_paragraphs | single_source | paragraph |
| `scierc_ner` | ie.named_entity_recognition | artificial_intelligence | paragraph | single_source | json |
| `scierc_re` | ie.relation_extraction | artificial_intelligence | paragraph | single_source | json |
| `scifact_entailment` | entailment | biomedicine, clinical_medicine | paragraph | single_source | json |
| `scireviewgen_multidoc_summarization` | summarization | artificial_intelligence | multiple_paragraphs | multiple_source | paragraph |
| `scitldr_aic` | summarization | artificial_intelligence | multiple_paragraphs | single_source | sentence |

搜集汇总

背景与挑战

背景概述

SciRIFF数据集包含13.7万条指令遵循演示，覆盖54个科学文献理解任务，涉及五个科学文献类别和领域。它提供三种不同最大上下文长度的配置，并基于Apache License 2.0和ODC-By许可发布，由allenai维护。