AllTheBacteria/BacCorpus-prot-90

---
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
dataset_info:
features:
- name: id
dtype: string
- name: sequence
dtype: string
splits:
- name: train
num_bytes: 315917457977
num_examples: 939820597
download_size: 272058980539
dataset_size: 315917457977
tags:
- protein
- genomics
- bacteria
- genomic-language-models
- microbiology
- protein-language-models
- plm
- amino-acids
- biology
size_categories:
- 100M<n<1B
---

# BacCorpus Protein 90

BacCorpus Protein 90 is a large-scale, sequence-level clustered dataset of bacterial protein sequences. It contains representative protein sequences clustered at 90% sequence identity and 90% coverage, derived from BacCorpus, a uniformly processed bacterial genome resource spanning millions of quality-controlled isolate genomes and metagenome-assembled genomes.

This dataset is intended for training and evaluating protein language models and bacterial genomic language models. Protein sequences capture much of the conserved functional signal in bacterial genomes and are useful for modelling gene function, protein composition, and genome-scale bacterial phenotypes.

## Dataset Description

- **Repository:** `AllTheBacteria/BacCorpus-prot-90`
- **Domain:** bacterial genomics
- **Data type:** protein sequences
- **Organisms:** bacteria
- **Format:** Parquet
- **Scale:** approximately 940M representative protein sequences
- **Clustering:** 90% sequence identity and 90% coverage
- **Primary use case:** pretraining bacterial protein and genomic language models

BacCorpus Protein 90 was built by combining genomes from several public bacterial genome resources, including MGnify [1], SPIRE [2], HRGM [3], GTDB [4], mOTUs DB [5], and AllTheBacteria [6], annotating the genomes, extracting predicted protein-coding sequences, and clustering them with MMSeqs2 (version `18.8cc5c`).

## Intended Uses

This dataset is intended for:

- pretraining protein language models on bacterial proteins;
- training bacterial genomic language models;
- studying bacterial protein sequence diversity;
- sampling representative bacterial proteins for downstream model training;
- evaluating sequence redundancy reduction strategies for bacterial genomics.

## How to Use

Because this dataset is large, we recommend streaming it rather than downloading it locally.

```python
from datasets import load_dataset

ds = load_dataset(
"AllTheBacteria/BacCorpus-prot-90",
split="train",
streaming=True,
)

example = next(iter(ds))
print(example.keys())
print(example)
```

## Citation:
```bibtex
TBD
```

## References

[1] Richardson, L. et al. MGnify: the microbiome sequence data analysis resource in 2023. *Nucleic Acids Research* 51(D1), D753–D759 (2023). doi:10.1093/nar/gkac1080

[2] Schmidt, T. S. B. et al. SPIRE: a Searchable, Planetary-scale mIcrobiome REsource. *Nucleic Acids Research* 52(D1), D777–D783 (2024). doi:10.1093/nar/gkad943

[3] Almeida, A. et al. A unified catalog of 204,938 reference genomes from the human gut microbiome. *Nature Biotechnology* 39, 105–114 (2021). doi:10.1038/s41587-020-0603-3

[4] Parks, D. H. et al. GTDB: an ongoing census of bacterial and archaeal diversity through a phylogenetically consistent, rank normalized and complete genome-based taxonomy. *Nucleic Acids Research* 50(D1), D785–D794 (2022). doi:10.1093/nar/gkab776

[5] Dmitrijeva, M. et al. The mOTUs online database provides web-accessible genomic context to taxonomic profiling of microbial communities. *Nucleic Acids Research* 53(D1), D797–D805 (2025). doi:10.1093/nar/gkae1004

[6] Hunt, M. et al. AllTheBacteria – all bacterial genomes assembled, available, and searchable. *bioRxiv* (2024). doi:10.1101/2024.03.08.584059