Auxiliary Datasets for Speaker Disambiguation in Quotebank

This data repository contains the auxiliary datasets necessary for enriching and preprocessing Quotebank, a large dataset of unique, speaker-attributed quotations. The scripts utilizing the data can be found in the quotebank-toolkit GitHub repository. The datasets are stored in <strong>data.zip </strong>and are described below: <strong>quotebank_disambiguation_mapping_quote.parquet</strong><br> Provides the `quoteID`-&gt; `speakerQID` mapping for each quotation, disambiguating ambiguous speaker names and linking them to their respective Wikidata items. The schema of the dataset is as follows: <pre><code> |-- quoteID: primary key of the quotation (format: "YYYY-MM-DD-{increasing int:06d}") |-- speaker: Wikidata ID corresponding to the speaker of the quotation</code></pre> The mapping is created using heuristics described in the following paper: Marko Čuljak, Andreas Spitz, Robert West, and Akhil Arora<br> "Strong Heuristics for Named Entity Linking"<br> <em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop</em><br> 10.18653/v1/2022.naacl-srw.30 <strong>self_quotations_filtered.parquet</strong><br> Contains the identifiers of the quotations identified as not being self-attributed. The schema of the dataset is as follows: <pre><code> |-- quoteID: primary key of the quotation (format: "YYYY-MM-DD-{increasing int:06d}") </code></pre> <strong>speaker_attributes.parquet</strong><br> Contains attributes of all the speakers appearing in Quotebank extracted from Wikidata. The schema of the dataset is as follows. <pre><code> |-- id: Wikidata item QID of the speaker, primary key |-- aliases: list of speaker's aliases |-- date_of_birth: list of possible speaker's dates of birth |-- nationality: list of speaker's nationalities |-- gender: list of speaker's previous or current genders |-- lastrevid: ID of the last revision of the speaker's item |-- ethnic_group: list of ethnic groups the speaker belongs to |-- US_congress_bio_ID: identifier for the speaker in the Biographical Directory of the United States Congress |-- occupation: list of speaker's occupations |-- party: list of parties the speaker is/was affiliated to |-- academic_degree: list of academic degrees obtained by the speaker |-- label: Wikidata label of the speaker |-- candidacy: list of the speaker's candidacies in political elections |-- type: type of the Wikidata entry (value is `item` for all the speakers) |-- religion: previous/current religious affiliations of the speaker </code></pre> Using the `id` field corresponding to the Wikidata QID of a speaker, this dataset can be easily joined with disambiguated Quotebank obtained by running the `cleanup_disambiguate.py` script available in the aforementioned quotebank-toolkit repository.