Adam

Adam has become the default method for managing step sizes in neural networks. It combines the ideas of Momentum (ML) and adadelta.

• Take the average of the gradient
• Moderate step size based on magnitude of gradient

We start by writing the moving averages of the gradient and squared gradient, which reflect estimates of the mean and variance of the gradient for weight $j$:

$$\begin{array}{rl}{g}_{t,j}& ={\nabla }_{W}J(W_{t-1}{)}_j\\ m_{t,j}& =B_1{m}_{t-1,j}+(1-B_1)g_{t,j}\\ v_{t,j}& =B_2{v}_{t-1,j}+(1-B_2)g_{t,j}^2\end{array}$$

A problem with these estimates is that, if we initialize $m_0=v_0=0$, they will always be biased (slightly too small). So we will correct for that bias by defining:

$$\begin{array}{rl}{\hat{m}}_{t,j}& =\frac{m_{t,j}}{1-B_1^t}\\ {\hat{v}}_{t,j}& =\frac{v_{t,j}}{1-B_2^t}\\ W_{t,j}& =W_{t-1,j}-\frac{\eta }{\sqrt{{\hat{v}}_{t,j}+ϵ}}{\hat{m}}_{t,j}\end{array}$$

Note that $B_1^t$ is $B_1$ raised to the power of $t$ and likewise for $B_2^t$. To justify these corrections, note that if we were to expand $m_{t,j}$ in terms of $m_{0,j}$ and $g_{0,j},g_{1,j},\dots ,g_{t,j}$ the coefficients would sum to $1.$ However, the coefficient behind $m_{0,j}$ is $B_1^t$ and since $m_{0,j}=0$, the sum of coefficients of non-zero terms is $1-B_1^t$ , hence the correction. The same justification holds for $v_{t,j}$.

Now, our update for weight $j$ has a step size that takes the steepness into account (like Adadelta) but also tends to move in the same direction (like momentum). The authors of adam like setting:

• $B_1=0.9$
• $B_2=0.990$
• $ϵ=10^{-8}$

Although we now have even more parameters, adam is not highly sensitive to their values. Even though we now have a step-size for each weight, and we have to update various quantities on each iteration of gradient descent, it’s relatively easy to implement by maintaining a matrix for each quantity $(m_t^l,v_t^l,g_t^l,g_t^{2^l})$ in each layer of the network.