Weight Decay

Weight decay is a simple regularization strategy that penalizes the norm of all weights, as we did in Ridge Regression. We take the gradient of the objective

$$J(W)=\sum _{i=1}^{n}L(x^{(i)},y^{(i)})+\lambda ||W|{|}^2$$

and end up with an update of the form

$$\begin{array}{rl}{W}_t& =W_{t-1}-\eta (({\nabla }_W\text{Loss}(\text{NN}(x^{(i)}),y^{(i)};W_{t-1}))+\lambda W_{t-1})\\ & =W_{t-1}(1-\lambda \eta )-\eta ({\nabla }_W\text{Loss}(\text{NN}(x^{(i)}),y^{(i)};W_{t-1}))\end{array}$$

So, we are “decaying” $W_{t-1}$ by a factor of $(1-\lambda \eta )$ and then taking a gradient step.

Weight decay seems to be very similar to L2 Regularization and is equivalent in some cases. I don’t understand the distinction yet.