UDL Chapter 9 Problems

Problem 9.1

Consider a model where the prior distribution over the parameter is a normal distribution with mean zero and variance ${\sigma }_{\varphi }^2$ so that

$$Pr(\varphi )=\prod _{j=1}^J{\text{Norm}}_{{\varphi }_j}[0,{\sigma }_{\varphi }^2]$$

where $j$ indexes the model parameters. We now maximize ${\sum }_{i=1}^{I}Pr({\mathbf{y}}_i|{\mathbf{x}}_i,\varphi )Pr(\varphi )$. Show that the associated loss function of this model is equivalent to L2 regularization.

Recall from Probabilistic interpretation that the regularization term can be consider a prior $Pr(\varphi )$ representing some knowledge we have about the parameters.

The posterior objective is then:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmax}}\left[\prod _{i=1}^{I}Pr({\mathbf{y}}_i|{\mathbf{x}}_i,\varphi )Pr(\varphi )\right]$$

We use the log to convert from product to sum:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmax}}\left[\sum _{i=1}^I\log Pr(y_i|x_i,\varphi )+\log Pr(\varphi )\right]$$

The prior term is:

$$\begin{array}{rl}\log Pr(\varphi )& =\log \prod _{j=1}^J\frac{1}{\sqrt{2\pi {\sigma }_{\varphi }^2}}\exp \left(-\frac{{\varphi }_j^2}{2{\sigma }_{\varphi }^2}\right)\\ & =\sum _{j=1}^J-\frac{1}{2}\log (2\pi {\sigma }_{\varphi }^2)-\frac{{\varphi }_j^2}{2{\sigma }_{\varphi }^2}\\ & =\sum _{j=1}^{J}C-\frac{{\varphi }_j^2}{2{\sigma }_{\varphi }^2}\end{array}$$

where we collapsed the first term into $C$ because it does not depend on the parameters $\varphi$.

Therefore, maximizing the log posterior is equivalent to

$$\begin{array}{rl}\hat{\varphi }& =\underset{\varphi }{\mathrm{argmax}}\left[\sum _{i=1}^I\log Pr(y_i\mid x_i,\varphi )-\frac{1}{2{\sigma }_{\varphi }^2}\sum _{j=1}^J{\varphi }_j^2\right]\\ & =\underset{\varphi }{\mathrm{argmax}}\sum _{i=1}^I\log Pr(y_i|x_i,\varphi )-\frac{1}{2{\sigma }_{\varphi }^2}||\varphi |{|}_2^2\end{array}$$

or equivalently minimizing

$$L(\varphi )=-\sum _{i=1}^I\log Pr(y_i|x_i,\varphi )+\lambda ||\varphi |{|}_2^2$$

with $\lambda =-\frac{1}{2{\sigma }_{\varphi }^2}$.

Problem 9.2

How do the gradients of the loss function change when L2 regularization is added?

The parameters are incentivized to stay small (near zero), as larger norm will cause the loss value to be higher.

$$L(\varphi )=L_0(\varphi )+\lambda ||\varphi |{|}^2$$

where $L_0(\varphi )$ is the regular NLL objective.

Then the gradient becomes:

$${\nabla }_{\varphi }L(\varphi )={\nabla }_{\varphi }{L}_0(\varphi )+2\lambda \varphi$$

Thus, every parameter is pulled toward zero proportionally to its magnitude.

Problem 3

Problem 4

Problem 5

Problem 6

Problem 7

Problem 8

Problem 9

Problem 10