Nesterov Accelerated Momentum

The momentum term can be considered a coarse prediction where the SGD algorithm will move next. Nesterov accelerated momentum computes the gradients at this predicted point rather than at the current point:

$$\begin{array}{rl}{\mathbf{m}}_{t+1}& ⟵\beta \cdot {\mathbf{m}}_t+(1-\beta )\sum _{i\in {\mathcal{B}}_t}\frac{\partial {\ell }_i[{\varphi }_t-\alpha \beta \cdot {\mathbf{m}}_t]}{\partial \varphi }\\ {\varphi }_{t+1}& ⟵{\varphi }_t-\alpha \cdot {\mathbf{m}}_{t+1}\end{array}$$

where now the gradients are evaluated at ${\varphi }_t-\alpha \beta \cdot {\mathbf{m}}_t$.

• One way to think about this that the gradient term now corrects the path provided by momentum alone.