This is the code and data for IS-RNN validation and research

Instructions:

# Importance Sampling for RNNs

1. Prerequisite

a. Install Theano

b. Nvidia GPU with memory > 8G

c. Install CUDA 9.0 and Cudnn > 7.0

d. Install Libgpuarray for theano

e. Install midi package for python (not in pip repo), we include this package in 'code' folder.

2. Run

a. ./run_all_cases.sh in 'code' folder for IS-RNN, IS-LSTM and IS-RNN-RBM.

b. ./run_nmt.sh in 'nmt_code' folder for IS-GRU-NMT.

3. Visualization

a. ./print_rnn.py, ./print_lstm.py and ./print_rbm.py in 'code' folder

b. ./print_nmt.py in 'nmt_code' folder

4. Misc

a. Change interval which controls the IS interval as you wish, we empirically set

it as 3 or 4. In fact, it can be larger and IS still achieves accelerated convergence rate.

b. We collect gradient norm and loss, score, etc at the end of every epoch for results presentation.

Be noted that the gradient norm calculated is only used to generate new IS sequence when epoch%interval==0.

In other epochs, the gradient norm is only calculated for results presentation.

Comment out this section and thus the gradient norm is actually calculated every interval epochs.

d. For IS-LSTM case, you need GPU memory >12G (we use 24G P40), if you encounter OOM, please consider

reduce batch-size or swith test (25,000) to train (1,998).

e. seq_gen (compiled from seq_gen.cpp) is responsible for sampling w.r.t P, which is called within python script.

Contact:

Advanced AI Lab, Shanghai Jiao Tong University

Fei Wang PhD@SJTU

Datasets

IS-RNN