CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot

The CREATE database is composed of 14 hours of multimodal recordings from a mobile robotic platform based on the iRobot Create. The dataset has been designed in the context of a mobile robot that can navigate and learn multimodal representations of its environment. The recordings include stereo RGB images, stereo audio, inertial measurement unit (accelerometer, gyroscope, magnetometer), odometry, battery current and much more. The multimodal dataset is expected to have multiple usages: 

  • Multimodal unsupervised object learning: learn the statistical regularities (structures) of the sensor inputs per modality and across modalities.
  • Multimodal prediction: learn to predict future states of the sensory inputs. 
  • Egomotion detection: learn to predict motor states from the other sensory inputs (e.g. visual optical flow, gyroscope).
  • Causality detection: learn to predict when the robot affects its own sensory inputs (i.e. due to motors), and when the environment is perturbing the sensory inputs (e.g. the user moves the robot around, the robot sees a human moving).

 

Instructions: 

Provided Files

  • CREATE-hdf5-e1.zip          :   HDF5 files for Experiment I
  • CREATE-hdf5-e2.zip          :   HDF5 files for Experiment II
  • CREATE-hdf5-e3.zip          :   HDF5 files for Experiment III
  • CREATE-preview.zip          :   Preview MP4 videos and PDF images
  • CREATE-doc-extra.zip       :   Documentation: CAD files, datasheets and images
  • CREATE-source-code.zip  :   Source code for recording, preprocessing and examples

 

Extract all ZIP archives in the same directory (e.g. $HOME/Data/Create).Examples of source code (MATLAB and Python) for loading and displaying the data are included.For more details about the dataset, see the specifications document in the documentation section. 

Dataset File Format

The data is provided as a set of HDF5 files, one per recording session. The files are named to include the location (room) and session identifiers, as well as the recording date and time (ISO 8601 format). The recording sessions related to a particular experiment are stored in a separate folder. Overall, the file hierarchy is as follows:

<EXP_ID>/<LOC_ID>/<EXP_ID>_<LOC_ID>_<SESS_ID>_<DATETIME>.h5

 

Summary of Available Sensors

The following sensors were recorded and made available in the CREATE dataset:

  • Left and right RGB cameras (320x240, JPEG, 30 Hz sampling rate)
  • Left and right optical flow fields (16x12 sparse grid, 30 Hz sampling rate)
  • Left and right microphones (16000 Hz sampling rate, 64 ms frame length)
  • Inertial measurement unit: accelerometer, gyroscope, magnetometer (90 Hz sampling rate)
  • Battery state (50 Hz sampling rate)
  • Left and right motor velocities (50 Hz sampling rate)
  • Infrared and contact sensors (50 Hz sampling rate)
  • Odometry (50 Hz sampling rate)
  • Atmospheric pressure (50 Hz sampling rate)
  • Air temperature (1 Hz sampling rate)

 

Other relevant information about the recordings is also included:

  • Room location, date and time of the session.
  • Stereo calibration parameters for the RGB cameras.

 

Summary of Experiments

Experiment I: Navigation in Passive Environments

The robot was moving around a room, controlled by the experimenter using a joystick. Each recorded session was approximately 15 min. There are 4 session recordings per room, with various starting points and trajectories. There was little to no moving objects (including humans) in the room. The robot was directed by the experimenter not to hit any obstacles. 

Experiment II: Navigation in Environments with Passive Human Interactions

In this experiment, the robot was moving around a room, controlled by the experimenter using a joystick. Each recorded session was approximately 15 min. There are 4 session recordings per room, with various starting points and trajectories. Note that compared to Experiment I, there was a significant amount of moving objects (including humans) in the selected rooms. 

Experiment III: Navigation in Environments with Active Human Interactions

The robot was moving around a room, controlled by the experimenter using a joystick. A second experimenter lifted the robot and changed its position and orientation at random intervals (e.g. once every 10 sec). Each recorded session was approximately 15 min. There are 5 session recordings in a single room. 

Acknowledgements

The authors would like to thank the ERA-NET (CHIST-ERA) and FRQNT organizations for funding this research as part of the European IGLU project.

 

Dataset Files

You must login with an IEEE Account to access these files. IEEE Accounts are FREE.

Sign Up now or login.

OPEN ACCESS Dataset Details

Citation Author(s):
Simon Brodeur, Simon Carrier, Jean Rouat
Submitted by:
Simon Brodeur
Last updated:
Thu, 02/15/2018 - 09:48
DOI:
10.21227/H2M94J
Data Format:
 
Cite

Documentation

AttachmentSize
PDF icon Detailed information10.72 MB
File Sample video8.76 MB

Subscribe

[1] Simon Brodeur, Simon Carrier, Jean Rouat, "CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot", IEEE Dataport, 2018. [Online]. Available: http://dx.doi.org/10.21227/H2M94J. Accessed: Feb. 19, 2018.
@data{h2m94j-18,
doi = {10.21227/H2M94J},
url = {http://dx.doi.org/10.21227/H2M94J},
author = {Simon Brodeur; Simon Carrier; Jean Rouat },
publisher = {IEEE Dataport},
title = {CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot},
year = {2018} }
TY - DATA
T1 - CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot
AU - Simon Brodeur; Simon Carrier; Jean Rouat
PY - 2018
PB - IEEE Dataport
UR - 10.21227/H2M94J
ER -
Simon Brodeur, Simon Carrier, Jean Rouat. (2018). CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot. IEEE Dataport. http://dx.doi.org/10.21227/H2M94J
Simon Brodeur, Simon Carrier, Jean Rouat, 2018. CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot. Available at: http://dx.doi.org/10.21227/H2M94J.
Simon Brodeur, Simon Carrier, Jean Rouat. (2018). "CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot." Web.
1. Simon Brodeur, Simon Carrier, Jean Rouat. CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot [Internet]. IEEE Dataport; 2018. Available from : http://dx.doi.org/10.21227/H2M94J
Simon Brodeur, Simon Carrier, Jean Rouat. "CREATE: Multimodal Dataset for Unsupervised Learning and Generative Modeling of Sensory Data from a Mobile Robot." doi: 10.21227/H2M94J