UDL Chapter 5 Problems

Problem 5.1

Show that the logistic sigmoid function $\text{sig}[z]$ becomes $0$ as $z\to -\infty$, is $0.5$ when $z=0$, and becomes $1$ when $z\to \infty$, where

$$\text{sig}[z]=\frac{1}{1+\exp [-z]}$$

For $z\to -\infty$:

$$\underset{z\to -\infty }{\lim }\frac{1}{1+\exp [-z]}=\frac{1}{\infty }=0$$

For $z=0$:

$$\frac{1}{1+\exp [0]}=\frac{1}{1+1}=\frac{1}{2}=0.5$$

For $z\to \infty$:

$$\underset{z\to -\infty }{\lim }\frac{1}{1+\exp [-z]}=\frac{1}{1+0}=1$$

Problem 5.2

The loss $L$ for binary classification for a single training pair $\{\mathbf{x},y\}$ is

$$L=-(1-y)\log [1-\text{sig}[f[\mathbf{x},\varphi ]]]-y\log [\text{sig}[f[\mathbf{x},\varphi ]]]$$

Plot this loss as a function of the transformed output $\text{sig}[f[\mathbf{x},\varphi ]]\in [0,1]$ (i) when the training label $y=0$ and when (ii) when $y=1$.

When $y=0$, we just have $L=-1\log [1-\text{sig}[f[x,\varphi ]]]$. With $y=1$, we just have $L=-\log [\text{sig}[f[x,\varphi ]]]$:

Problem 5.3

Suppose we want to build a model that predicts the direction $y$ in radians of the prevailing wind based on local measurements of barometric pressure $\mathbf{x}$. A suitable distribution over circular domains is the von Mises distribution:

$$Pr(y|\mu ,\kappa )=\frac{\exp [\kappa \cos [y-\mu ]]}{2\pi \cdot {\text{Bessel}}_0[\kappa ]}$$

• $\mu$ is a measure of the mean direction

• $\kappa$ is a measure of concentration (i.e. inverse of variance)

• The term ${\text{Bessel}}_0(\kappa )$ is a modified Bessel function of the first kind of order $0$.

  Use the loss function recipe to develop a loss function for learning the parameter $\mu$ of a model $f[\mathbf{x},\varphi ]$ to predict the most likely wind direction. Your solution should treat the concentration $\kappa$ as a constant. How would you perform inference?

We set $\mu =f[{\mathbf{x}}_i,\varphi ]$, so

$$Pr(y|\mu ,\kappa )=\frac{\exp [\kappa \cos [y-f[{\mathbf{x}}_i,\varphi ]]]}{2\pi \cdot {\text{Bessel}}_0[\kappa ]}$$

Then the negative log-likelihood loss function is

$$\begin{array}{rl}L& =-\sum _{i=1}^I\log \left(\frac{\exp [\kappa \cos [y-f[{\mathbf{x}}_i,\varphi ]]]}{2\pi \cdot {\text{Bessel}}_0[\kappa ]}\right)\\ & =-\sum _{i=1}^I[\kappa \cos [y-f[{\mathbf{x}}_i,\varphi ]]]-(\log (2\pi )+\log ({\text{Bessel}}_0[\kappa ]))\\ & =-\sum _{i=1}^I\cos [y-f[{\mathbf{x}}_i,\varphi ]]\end{array}$$

To perform inference we just take the maximum of the distribution (which is just the predicted parameter $\mu$). This might be out of the range $[-\pi ,\pi ]$, in which case we would add/remove multiples of $2\pi$ until it is in the right range.

Problem 5.4

Sometimes, the outputs for input $\mathbf{x}$ are multimodal; there is more than one valid prediction for a given input. Here, we might use a sum of normal components as the distribution over the output. This is known as a mixture of Gaussians model. For example, a mixture of two Gaussians has parameters $\theta =\{\lambda ,{\mu }_1,{\sigma }_1^2,{\mu }_2,{\sigma }_2^2\}$:

$$Pr(y|\lambda ,{\mu }_1,{\mu }_2,{\sigma }_1^2,{\sigma }_2^2)=\frac{\lambda }{\sqrt{2\pi {\sigma }_1^2}}\exp \left[-\frac{(y-{\mu }_1{)}^2}{2{\sigma }_1^2}\right]+\frac{1-\lambda }{\sqrt{2\pi {\sigma }_2^2}}\exp \left[\frac{-(y-{\mu }_2{)}^2}{2{\sigma }_2^2}\right]$$

where $\lambda \in [0,1]$ controls the relative weight of the two components, which have means ${\mu }_1,{\mu }_2$ and variances ${\sigma }_1^2,{\sigma }_2^2$, respectively. This model can represent a distribution with two peaks or a distribution with one peak but a more complex shape.

Use the loss function recipe to construct a loss function for training a model $\mathbf{f}[x,\varphi ]$ that takes input $x$, has parameters $\varphi$, and predicts a mixture of two Gaussians. The loss should be based on $I$ training data pairs $\{x_i,y_i\}$. What problems do you foresee when performing inference?

Let:

• $\lambda ={\mathbf{f}}_1[{\mathbf{x}}_i,\varphi ]$
• ${\mu }_1={\mathbf{f}}_2[{\mathbf{x}}_i,\varphi ]$
• ${\sigma }_1^2={\mathbf{f}}_3[{\mathbf{x}}_i,\varphi ]$
• ${\mu }_2={\mathbf{f}}_4[{\mathbf{x}}_i,\varphi ]$
• ${\sigma }_2^2={\mathbf{f}}_5[{\mathbf{x}}_i,\varphi ]$

Then the loss is

$$L=-\sum _{i=1}^I\log \left[\frac{\text{sig}[{\mathbf{f}}_1[{\mathbf{x}}_i,\varphi ]]}{\sqrt{2\pi {\mathbf{f}}_3[{\mathbf{x}}_i,\varphi ]}}\exp \left[\frac{-(y-{\mathbf{f}}_2[{\mathbf{x}}_i,\varphi ]{)}^2}{2{\mathbf{f}}_3[{\mathbf{x}}_i,\varphi ]}\right]+\frac{1-\text{sig}[{\mathbf{f}}_1[{\mathbf{x}}_i,\varphi ]]}{\sqrt{2\pi {\mathbf{f}}_5[{\mathbf{x}}_i,\varphi ]}}\exp \left[\frac{-(y-{\mathbf{f}}_4[{\mathbf{x}}_i,\varphi ]{)}^2}{2{\mathbf{f}}_5[{\mathbf{x}}_i,\varphi ]}\right]\right]$$

Inference is a bit trickier in this case since there is no simple closed form for the mode of this distribution.

• MACP: How many modes can a Gaussian mixture have?

Problem 5.5

Consider extending the model from problem 5.3 to predict the wind direction using a mixture of two von Mises distributions. Write an expression for the likelihood $Pr(y|\theta )$ for this model. How many outputs will the network produce?

Each von Mises distribution is parametrized by $\mu ,\kappa$. Thus, for a mixture of two von Mises distributions, the parameters will be

$$\theta =\{\lambda ,{\mu }_1,{\kappa }_1,{\mu }_2,{\kappa }_2\}$$

where $\lambda$ is the relative weight of the two distributions. The likelihood will then be:

$$Pr(y|\lambda ,{\mu }_1,{\kappa }_1,{\mu }_2,{\kappa }_2)=\lambda \frac{\exp [{\kappa }_1\cos [y-{\mu }_1]}{2\pi \cdot {\text{Bessel}}_0[{\kappa }_1]}+(1-\lambda )\frac{[\exp [{\kappa }_2\cos [y-{\mu }_2]]}{2\pi \cdot {\text{Bessel}}_0[{\kappa }_2]}$$

Like the mixture of Gaussians above, we would need five outputs, unless we consider ${\kappa }_1$ and ${\kappa }_2$ to be constants, in which case we would need 3.

Problem 5.6

Consider building a model to predict the number of pedestrians $y\in \{0,1,2,\dots \}$ that will pass a given point in the city in the next minute, based on data $\mathbf{x}$ that contains information about the time of the day, the longitude and latitude, and the type of neighborhood. A suitable distribution for modeling counts is the Poisson distribution. This has a single parameter $\lambda >0$ called the rate that represents the mean of the distri

Problem 5.7

Problem 5.8

Problem 5.9

Problem 5.10