cooper-hewitt-alt-text

# Cooper Hewitt — Visual Descriptions with CC0 Images **49,253 museum objects**, each with a **human-written visual description** and a **CC0 image**. Openly-licensed, human-written, *pixel-grounded* descriptions of cultural-heritage images are close to non-existent. We swept Zenodo, the Hugging Face Hub (356 image-to-text/VQA datasets) and GitHub looking for them and came up essentially empty — `wikimedia/wit_base` has a real alt-text field but it is populated on only 4.8% of English rows at a **5-word median** ("Municipal office", "Flag of Aga"). This dataset exists because that gap is real. Everything here is **CC0 end to end** — image *and* text — so derivatives can be republished. ## What a row looks like >  > > **Beach at Low Tide, Cullercoats, England** — Winslow Homer, 1881, charcoal on paper > > *"Horizontal view of a beach at low tide with small boats standing in to port and one boat > being beached with a carriage."* ## Fields `image` (CC0, ≤1024px) · `description` · `title` · `maker` · `date` · `medium` · `object_type` · `record_id` · `record_link` · `image_url` · `ids_id` **Quality flags — we flag, we do not filter.** Slice it yourself: | Flag | Meaning | Count | |---|---|---| | `n_words` / `in_target_band` | 16–30 words | 15,264 | | `n_proper_nouns` | proxy for iconographic leakage (see below) | 18,557 rows ≥1 | | `has_artifact` | a dropped word leaving a double space | 4,564 | | `has_format_prefix` | the `"Horizontal rectangle."` convention | 2,973 | | `has_crossref` | cataloguer shorthand (`"a variation of -1638"`) | 1,676 | A reasonable default: `n_words >= 8 and not has_crossref and not has_artifact` → **40,541 rows**. For a stricter caption seed, add `in_target_band and n_proper_nouns == 0`. ## Read this before you train on it **These are not purpose-written alt-text.** The source field is `extDescrAccessibility` in Smithsonian Open Access — but inspect the record and you find the identical string in `freetext.notes` as `"Description: …"`. It is the museum's **catalogue description note, copied into the accessibility slot**. That origin explains everything below. **It leaks iconography.** Giani's *Venus and Cupid and Satyr* (`chndm_1901-39-3506`) is described as *"Antiope reclines at center, apparently asleep."* **Antiope** is not derivable from pixels — a cataloguer knows the Correggio source. Train on this and you teach a model to confidently name figures it cannot see, which is precisely the *factual misrecognition* failure mode documented for VLMs on heritage images. `n_proper_nouns` exists so you can avoid it: **62% of rows contain no proper noun at all.** **It omits.** The Winslow Homer above never mentions the large dark **foreground figure standing in the water** — the most salient thing in the picture. A cataloguer is not writing for a screen-reader user. Expect *selective omission* in the ground truth itself. **It is a work in progress.** Records carry `Catalogue Status: Research in Progress` upstream. So: a strong, large, CC0 **seed** — not gold-standard alt-text. Human rating still has real work to do on top of it. ## Provenance - Metadata: [Smithsonian Open Access](https://www.si.edu/openaccess), unit **CHNDM** (`s3://smithsonian-open-access/metadata/edan/chndm/`). CC0. - Images: Smithsonian IDS (`ids.si.edu`), fetched at `max=1024`. Every row's first media object carries `usage.access == "CC0"` — that is the inclusion criterion. - Museum: [Cooper Hewitt, Smithsonian Design Museum](https://collection.cooperhewitt.org). Gotcha for anyone re-harvesting: the EDAN shards are **hex-named** (`00.txt`–`ff.txt`). Iterating `0..99` in decimal silently fetches 100 of 256 shards, returns HTTP 200 throughout, and loses ~60% of the corpus (19,274 rows instead of 49,253). Please credit Cooper Hewitt, Smithsonian Design Museum.