Optimizing Neural Networks

An optimizer takes in a gradient and decides how to update the parameters based on the gradients. The simplest form is Stochastic Gradient Descent.

• Adaptive step-size
  • Running Averages
  • Momentum (ML)
• Adadelta
• Adaptive moment estimation