IS-RNN

IS-RNN

Citation Author(s):
Fei Wang
Submitted by:
Fei Wang
Last updated:
Thu, 11/08/2018 - 10:34
DOI:
10.21227/H2V953
Data Format:
License:
Dataset Views:
78
Share / Embed Cite
Abstract: 

This is the code and data for IS-RNN validation and research

Instructions: 

# Importance Sampling for RNNs

 

1. Prerequisite

 

a. Install Theano

b. Nvidia GPU with memory > 8G

c. Install CUDA 9.0 and Cudnn > 7.0

d. Install Libgpuarray for theano

e. Install midi package for python (not in pip repo), we include this package in 'code' folder.

 

2. Run

   

    a. ./run_all_cases.sh in 'code' folder for IS-RNN, IS-LSTM and IS-RNN-RBM.

    b. ./run_nmt.sh in 'nmt_code' folder for IS-GRU-NMT.

 

3. Visualization

 

    a. ./print_rnn.py, ./print_lstm.py and ./print_rbm.py in 'code' folder

    b. ./print_nmt.py in 'nmt_code' folder

 

4. Misc

 

    a. Change interval which controls the IS interval as you wish, we empirically set 

       it as 3 or 4. In fact, it can be larger and IS still achieves accelerated convergence rate.

 

    b. We collect gradient norm and loss, score, etc at the end of every epoch for results presentation.

       Be noted that the gradient norm calculated is only used to generate new IS sequence when epoch%interval==0.

       In other epochs, the gradient norm is only calculated for results presentation.

       Comment out this section and thus the gradient norm is actually calculated every interval epochs.

 

    c. Update and optimization can be find on https://github.com/FayW/ImportanceSamplingRNN.git

 

    d. For IS-LSTM case, you need GPU memory >12G (we use 24G P40), if you encounter OOM, please consider

       reduce batch-size or swith test (25,000) to train (1,998).

 

    e. seq_gen (compiled from seq_gen.cpp) is responsible for sampling w.r.t P, which is called within python script.

 

Contact:

 

    Advanced AI Lab, Shanghai Jiao Tong University

 

    Fei Wang PhD@SJTU 

 

    fay96816@gmail.com

 

Dataset Files

You must be an IEEE Dataport Subscriber to access these files. Login or subscribe now. Sign up to be a Beta Tester and receive a coupon code for a free subscription to IEEE DataPort!

Embed this dataset on another website

Copy and paste the HTML code below to embed your dataset:

Share via email or social media

Click the buttons below:

facebooktwittermailshare
[1] Fei Wang, "IS-RNN", IEEE Dataport, 2018. [Online]. Available: http://dx.doi.org/10.21227/H2V953. Accessed: Dec. 09, 2019.
@data{h2v953-18,
doi = {10.21227/H2V953},
url = {http://dx.doi.org/10.21227/H2V953},
author = {Fei Wang },
publisher = {IEEE Dataport},
title = {IS-RNN},
year = {2018} }
TY - DATA
T1 - IS-RNN
AU - Fei Wang
PY - 2018
PB - IEEE Dataport
UR - 10.21227/H2V953
ER -
Fei Wang. (2018). IS-RNN. IEEE Dataport. http://dx.doi.org/10.21227/H2V953
Fei Wang, 2018. IS-RNN. Available at: http://dx.doi.org/10.21227/H2V953.
Fei Wang. (2018). "IS-RNN." Web.
1. Fei Wang. IS-RNN [Internet]. IEEE Dataport; 2018. Available from : http://dx.doi.org/10.21227/H2V953
Fei Wang. "IS-RNN." doi: 10.21227/H2V953