Accessing IEEE DataPort Dataset Files through Amazon Web Services (AWS)

Background

• IEEE DataPort dataset files are stored on Amazon Simple Storage Service (S3) file storage
• To find IEEE DataPort files and metadata, please see the Instructions for Accessing a Dataset found on the IEEE DataPort site.
• Users may download dataset files or access these files “in the cloud” using tools and services that support the AWS S3 protocol (see below for examples and links).  These instructions address accessing IEEE DataPort datasets files through AWS. 

Prerequisites

• User must have an IEEE account.   If you do not have an IEEE account, you may register for a free IEEE account (click this link to register).
• You must be an IEEE DataPort subscriber. You may subscribe to IEEE DataPort here.

Instructions

1. Login to IEEE DataPort (the login link is located in the upper right of the website)
2. Click on the Account link on the upper right
3. Find and record your personal AWS Access Key ID.
4. Click the Show AWS Secret Access Key link.
5. Find and record the AWS Secret Access Key, which will be shown in a dialog.
6. Now find the dataset you would like to access or analyze. See our Instructions for Accessing a Dataset for more information.
7. Copy the S3 URIs of the dataset files you wish to access.
   1. When you click on individual files under the AWS S3 tab, the S3 URI will be automatically selected. You may then copy (Ctrl-C, Cmd-C) and paste the URI.
   2. To copy all of the URIs for your dataset you may also click the Copy All URIs to the Clipboard link at the bottom of the AWS S3 tab. Note: some browsers do not support this feature, in which case a dialog is opened with all S3 URI’s listed and highlighted for easy copying.
8. Using your AWS Keys and the S3 URIs you copied, you will now be able to access these files using a number of tools, scripts, and services that support AWS S3. 
9. See below for links to some of the methods of accessing S3 files using your AWS keys. Note: the appropriate tool/service/script is dependent on the format of the dataset files you are trying to access. Instructions available on the dataset page may be helpful in selecting the appropriate tools.
    

Example using AWS CLI

Prerequisite:  Install the AWS CLI (v2) using the installer for your OS:  https://aws.amazon.com/cli/

Step 1.  Note your AWS Access Key and Secret Access Key associated with your IEEE DataPort Account. You can find this information by logging into IEEE DataPort and then clicking My Account on the upper right.

Step 2.  Then from the command line enter your Access/Secret Key:

$ aws configure set aws_access_key_id <your-access-key-here>

$ aws configure set aws_secret_access_key <your-secret-key-here>

Optionally you may verify your identity with this command (returning a json array with your AWS account info):

$ aws sts get-caller-identity

Step 3. Now you can download the file. To download the file into your current directory, enter the following command:

$ aws s3 cp s3://ieee-dataport/<your-path>/<your-file-name> .

The dot at the end indicates the file is to be saved with the original name in the current directory. A different file name/directory may be specified.

You should see the following output:

download: ss3://ieee-dataport/<your-path>/<your-file-name> to .//<your-file-name>
 

Links to Tools and Services supporting S3

• Amazon Big Data Services
• Amazon Elastic Map Reduce (EMR)
  • See how to Import files with Distributed Cache for EMR 
• Amazon Red Shift
• See Redshift Credentials section for using your AWS keys to access data
• See Configuring the AWS Command Line Interface for information on using your AWS keys
• Amazon Command Line Interface
• Amazon QuickSight
• Developer tools for S3
• Programming with AWS:
• Python (SDK)
• Java (SDK)
• R (tutorial)
• S3 REST API