Tips for Training Recurrent Neural Networks

Some practical tricks for training recurrent neural networks:

Optimization Setup

Network Structure

Model Parameters

I hope this collection of tips is helpful. Please feel free to post further suggestions for the list and ask questions.

You can use this post under the open CC BY-SA 3.0 license and cite it as:

@misc{hafner2017rnntips,
  author = {Hafner, Danijar},
  title = {Tips for Training Recurrent Neural Networks},
  year = {2017},
  howpublished = {Blog post},
  url = {https://danijar.com/tips-for-training-recurrent-neural-networks/}
}