Momentum (ML)

We can use methods like Running Averages to describe strategies for computing $\eta$. Momentum is a simple method that does this by “averaging” recent gradient updates, so that if they have been bouncing back and forth in some direction, we take out that component of the motion. For momentum, we have:

$$\begin{array}{rl}{V}_0& =0\\ V_t& =\gamma V_{t-1}+\eta {\nabla }_{W}J(W_{t-1})\\ W_t& =W_{t-1}-V_t\end{array}$$

This can also be written as:

$$\begin{array}{rl}{M}_0& =0\\ M_t& =\gamma M_{t-1}+(1-\gamma ){\nabla }_{W}J(W_{t-1})\\ W_t& =W_{t-1}-{\eta }^{\prime }{M}_t\end{array}$$

These two forms are equivalent if we let $\eta ={\eta }^{\prime }(1-\gamma )$. Essentially, $V_t$ (or $M_t$) is a moving average of the gradient. We are doing an update with step size ${\eta }^{\prime }$ on a moving average of the gradients with parameter $\gamma$.

$V_t$ will be bigger in dimensions that consistently have the same sign for ${\nabla }_W$ and smaller for those that don’t. Of course we now have two parameters to set ($\eta$ and $\gamma$), but the hope is that the algorithm will perform better overall, so it will be worth trying to find good values for them. Often γ is set to be something like 0.9.