Synthetically Spoken STAIR

This dataset consists of synthetically spoken captions for the STAIR dataset. Following the same methodology as Chrupała <em>et al.</em> (see article | dataset | code) we generated speech for each caption of the STAIR dataset using Google's Text-to-Speech API. This dataset was used for visually grounded speech experiments (see article accepted at ICASSP2019). <pre><code>@INPROCEEDINGS{8683069, author={W. N. {Havard} and J. {Chevrot} and L. {Besacier}}, booktitle={ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, title={Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese}, year={2019}, volume={}, number={}, pages={8618-8622}, keywords={information retrieval;natural language processing;neural nets;speech processing;word processing;artificial neural attention;human attention;monolingual models;part-of-speech tags;nouns;neural models;visually grounded speech signal;English language;Japanese language;word endings;cross-lingual speech-to-speech retrieval;grounded language learning;attention mechanism;cross-lingual speech retrieval;recurrent neural networks.}, doi={10.1109/ICASSP.2019.8683069}, ISSN={2379-190X}, month={May},}</code></pre> The dataset comprises the following files : <strong>mp3-stair.tar.gz</strong> :<strong> </strong>MP3 files of each caption in the STAIR dataset. Filenames have the following pattern <em>imageID_captionID</em>, where both <em>imageID</em> and <em>captionID </em>correspond to those provided in the original dataset (see annotation format here) <strong>dataset.mfcc.npy</strong> : Numpy array with MFCC vectors for each caption. MFCC were extracted using python_speech_features with default configuration. To know to which caption the MFCC vectors belong to, you can use the files dataset.words.txt and dataset.ids.txt. <strong>dataset.words.txt</strong> : Captions corresponding to each MFCC vector (line number = position in Numpy array, starting from 0) <strong>dataset.ids.txt </strong>: IDs of the captions (<em>imageID_captionID</em>) corresponding to each MFCC vector (line number = position in Numpy array, starting from 0) Splits test <strong>test.txt </strong>: captions comprising the test split <strong>test_ids.txt</strong>: IDs of the captions in the test split <strong>test_tagged.txt </strong>: tagged version of the test split <strong>test-alignments.json.zip </strong>: Forced alignments of all the captions in the test split. (dictionary where the key corresponds to the caption ID in the STAIR dataset). <em>Due to an unknown error during upload, the JSON file had to be zipped...</em> train <strong>train.txt </strong>: captions comprising the train split <strong>train_ids.txt </strong>: IDs of the captions in the train split <strong>train_tagged.txt </strong>: tagged version of the train split val <strong>val.txt </strong>: captions comprising the val split <strong>val_ids.txt </strong>: IDs of the captions in the val split <strong>val_tagged.txt </strong>: tagged version of the val split