AnonAstroData/SBI-16-2D

---
license: cc-by-4.0
pretty_name: Raw space-based images from the Hubble Space Telescope
tags:
- astronomy
- compression
- images
---

# SBI-16-2D Dataset

SBI-16-2D is a dataset which is part of the AstroCompress project. It contains imaging data assembled from the Hubble Space Telescope (HST). <TODO>Describe data format</TODO>

# Usage

You first need to install the `datasets` and `astropy` packages:

```bash
pip install datasets astropy
```

There are two datasets: `tiny` and `full`, each with `train` and `test` splits. The `tiny` dataset has 2 2D images in the `train` and 1 in the `test`. The `full` dataset contains all the images in the `data/` directory.

## Local Use (RECOMMENDED)

You can clone this repo and use directly without connecting to hf:

```bash
git clone https://huggingface.co/datasets/AnonAstroData/SBI-16-2D
```

To pull all data files:
```
git lfs pull
```

Then `cd SBI-16-3D` and start python like:

```python
from datasets import load_dataset
dataset = load_dataset("./SBI-16-2D.py", "tiny", data_dir="./data/", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")
```

Now you should be able to use the `ds` variable like:

```python
ds["test"][0]["image"].shape # -> (TBD)
```

Note of course that it will take a long time to download and convert the images in the local cache for the `full` dataset. Afterward, the usage should be quick as the files are memory-mapped from disk.

## Use from Huggingface Directly

This method may only be an option when trying to access the "tiny" version of the dataset.

To directly use from this data from Huggingface, you'll want to log in on the command line before starting python:

```bash
huggingface-cli login
```

or

```
import huggingface_hub
huggingface_hub.login(token=token)
```

Then in your python script:

```python
from datasets import load_dataset
dataset = load_dataset("AstroCompress/SBI-16-2D", "tiny", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")
```

## Demo Colab Notebook
We provide a demo collab notebook to get started on using the dataset [here](https://colab.research.google.com/drive/1wcz7qMqSAMST2kXFlL-TbwpYR26gYIYy?usp=sharing).

## Utils scripts
Note that utils scripts such as `eval_baselines.py` must be run from the parent directory of `utils`, i.e. `python utils/eval_baselines.py`.

---
许可证：CC-BY-4.0
展示名称：哈勃空间望远镜原始天基图像
标签：
- 天文学
- 数据压缩
- 图像
---

# SBI-16-2D 数据集

SBI-16-2D 是隶属于AstroCompress项目的数据集，其包含由哈勃空间望远镜（Hubble Space Telescope, HST）采集整理的成像数据。<待补充：数据格式描述>

# 使用方法

首先需安装`datasets`与`astropy`两个Python库：
bash
pip install datasets astropy

本数据集包含`tiny`与`full`两个子数据集，二者均设有`train`（训练集）与`test`（测试集）划分。其中`tiny`子数据集的训练集包含2张二维图像，测试集包含1张；`full`子数据集则包含`data/`目录下的全部图像。

## 本地使用（推荐）

你可克隆本仓库并直接使用，无需连接Hugging Face平台：
bash
git clone https://huggingface.co/datasets/AnonAstroData/SBI-16-2D

如需拉取全部数据文件：

git lfs pull

随后执行`cd SBI-16-2D`（原文疑似笔误为`SBI-16-3D`）并启动Python，执行如下代码：
python
from datasets import load_dataset
dataset = load_dataset("./SBI-16-2D.py", "tiny", data_dir="./data/", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")

此时即可通过`ds`变量使用数据集，例如：
python
ds["test"][0]["image"].shape # -> (待确定)

需注意，加载`full`子数据集时，下载并转换本地缓存中的图像会耗费较长时间；完成后，由于文件采用磁盘内存映射方式加载，后续使用将极为高效。

## 直接通过Hugging Face平台使用

该方法仅适用于访问`tiny`子数据集的场景。

若要直接通过Hugging Face平台加载数据集，需在启动Python前通过命令行完成登录：
bash
huggingface-cli login

或
python
import huggingface_hub
huggingface_hub.login(token=token)

随后在Python脚本中执行如下代码：
python
from datasets import load_dataset
dataset = load_dataset("AstroCompress/SBI-16-2D", "tiny", writer_batch_size=1, trust_remote_code=True)
ds = dataset.with_format("np")

## Colab演示笔记本

我们提供了一份用于上手该数据集的Colab演示笔记本，[点击此处](https://colab.research.google.com/drive/1wcz7qMqSAMST2kXFlL-TbwpYR26gYIYy?usp=sharing)即可访问。

## 工具脚本

需注意，诸如`eval_baselines.py`这类工具脚本需从`utils`目录的上级目录执行，例如执行`python utils/eval_baselines.py`。