UDL Chapter 7 Problems

Problem 7.1

A two-layer network with two hidden units in each layer can be defined as:

$$\begin{array}{rl}y={\varphi }_0& +{\varphi }_{1}a[{\psi }_{01}+{\psi }_{11}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{21}a[{\theta }_{02}+{\theta }_{12}x]]\\ & +{\varphi }_{2}a[{\psi }_{02}+{\psi }_{12}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{22}a[{\theta }_{02}+{\theta }_{12}x]]\end{array}$$

where the functions $a[\bullet ]$ are ReLU functions. Compute the derivatives of the output $y$ with respect to each of the 13 parameters ${\varphi }_{\bullet },{\theta }_{\bullet \bullet }$ and ${\psi }_{\bullet \bullet }$ directly. The derivatives of the ReLU function with respect to its input $\partial a[z]/\partial [z]$ is the indicator function $\mathbb{I}[z>0]$, which returns one if the argument is greater than zero and zero otherwise.

Output layer:

$$\begin{array}{rl}\frac{\partial y}{\partial {\varphi }_0}& =1\\ \frac{\partial y}{\partial {\varphi }_1}& =a[{\psi }_{01}+{\psi }_{11}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{21}a[{\theta }_{02}+{\theta }_{12}x]]\\ \frac{\partial y}{\partial {\varphi }_2}& =a[{\psi }_{02}+{\psi }_{12}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{22}a[{\theta }_{02}+{\theta }_{12}x]]\end{array}$$

Second hidden layer: Let us first define

$$\begin{array}{r}{h}_{21}={\psi }_{01}+{\psi }_{11}\cdot a[z_1]+{\psi }_{21}\cdot a[z_2]\\ h_{22}={\psi }_{02}+{\psi }_{12}\cdot a[z_1]+{\psi }_{22}\cdot a[z_2]\end{array}$$

Then:

$$\begin{array}{rl}\frac{\partial y}{\partial {\psi }_{01}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{01}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot 1\\ \frac{\partial y}{\partial {\psi }_{11}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{11}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot a[{\theta }_{01}+{\theta }_{11}x]\\ \frac{\partial y}{\partial {\psi }_{21}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{21}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot a[{\theta }_{02}+{\theta }_{12}x]\\ \frac{\partial y}{\partial {\psi }_{02}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{02}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot 1\\ \frac{\partial y}{\partial {\psi }_{12}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{12}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot a[{\theta }_{01}+{\theta }_{11}x]\\ \frac{\partial y}{\partial {\psi }_{22}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{22}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot a[{\theta }_{02}+{\theta }_{12}x]\end{array}$$

First hidden layer: Let us first define

$$\begin{array}{r}{h}_{11}={\theta }_{01}+{\theta }_{11}x\\ h_{12}={\theta }_{02}+{\theta }_{12}x\end{array}$$

Then we have:

$$\begin{array}{rl}\frac{\partial y}{\partial {\theta }_{01}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial h_{21}}{\partial a[h_{11}]}\frac{\partial a[h_{11}]}{\partial h_{11}}\frac{\partial h_{11}}{\partial {\theta }_{01}}\\ & ={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot {\psi }_{11}\cdot \mathbb{I}(h_{11}>0)\cdot 1\\ \frac{\partial y}{\partial {\theta }_{11}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial h_{21}}{\partial a[h_{11}]}\frac{\partial a[h_{11}]}{\partial h_{11}}\frac{\partial h_{11}}{\partial {\theta }_{11}}\\ & ={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot {\psi }_{11}\cdot \mathbb{I}(h_{11}>0)\cdot x\\ \frac{\partial y}{\partial {\theta }_{02}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial h_{22}}{\partial a[h_{12}]}\frac{\partial a[h_{12}]}{\partial h_{12}}\frac{\partial h_{12}}{\partial {\theta }_{02}}\\ & ={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot {\psi }_{12}\cdot \mathbb{I}(h_{12}>0)\cdot 1\\ \frac{\partial y}{\partial {\theta }_{12}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial h_{22}}{\partial a[h_{12}]}\frac{\partial a[h_{12}]}{\partial h_{12}}\frac{\partial h_{12}}{\partial {\theta }_{12}}\\ & ={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot {\psi }_{12}\cdot \mathbb{I}(h_{12}>0)\cdot x\end{array}$$

Problem 7.2

Find an expression for the final term in each of the five chains of derivatives in equation 7.13.

$$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial f_2}& =\left(\frac{\partial {\ell }_i}{\partial f_3}\frac{\partial f_3}{\partial h_3}\right)\frac{\partial h_3}{\partial f_2}\\ & =2(f_3-y_i)\cdot {\omega }_3\cdot -\sin f_2\end{array}$$ $$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial h_2}& =\left(\frac{\partial {\ell }_i}{\partial f_3}\frac{\partial f_3}{\partial h_3}\frac{\partial h_3}{\partial f_2}\right)\frac{\partial f_2}{\partial h_2}\\ & =2(f_3-y_i)\cdot {\omega }_3\cdot -\sin f_2\cdot {\omega }_2\end{array}$$ $$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial f_1}& =\left(\frac{\partial {\ell }_i}{\partial f_3}\frac{\partial f_3}{\partial h_3}\frac{\partial h_3}{\partial f_2}\frac{\partial f_2}{\partial h_2}\right)\frac{\partial h_2}{\partial f_1}\\ & =2(f_3-y_i)\cdot {\omega }_3\cdot -\sin f_2\cdot {\omega }_2\cdot \exp [f_1]\end{array}$$ $$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial h_1}& =\left(\frac{\partial {\ell }_i}{\partial f_3}\frac{\partial f_3}{\partial h_3}\frac{\partial h_3}{\partial f_2}\frac{\partial f_2}{\partial h_2}\frac{\partial h_2}{\partial f_1}\right)\frac{\partial f_1}{\partial h_1}\\ & =2(f_3-y_i)\cdot {\omega }_3\cdot -\sin f_2\cdot {\omega }_2\cdot \exp [f_1]\cdot {\omega }_1\end{array}$$ $$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial f_0}& =\left(\frac{\partial {\ell }_i}{\partial f_3}\frac{\partial f_3}{\partial h_3}\frac{\partial h_3}{\partial f_2}\frac{\partial f_2}{\partial h_2}\frac{\partial h_2}{\partial f_1}\frac{\partial f_1}{\partial h_1}\right)\frac{\partial h_1}{\partial f_0}\\ & =2(f_3-y_i)\cdot {\omega }_3\cdot -\sin f_2\cdot {\omega }_2\cdot \exp [f_1]\cdot {\omega }_1\cdot \cos [f_0]\end{array}$$

Problem 7.3

What size are each of the terms in equation 7.20?

• $\frac{\partial {\mathbf{f}}_2}{\partial {\mathbf{h}}_2}$ is $D_3\times D_2$
• $\frac{\partial {\mathbf{h}}_2}{\partial {\mathbf{f}}_1}$ is $D_2\times D_2$
• $\frac{\partial {\mathbf{f}}_1}{\partial {\mathbf{h}}_1}$ is $D_2\times D_1$
• $\frac{\partial {\mathbf{h}}_1}{\partial {\mathbf{f}}_0}$is $D_1\times D_1$

Problem 7.4

Calculate the derivative $\frac{\partial {\ell }_i}{\partial \text{f}[{\mathbf{x}}_i,\varphi ]}$ for the least squares loss function:

$${\ell }_i=(y_i-\text{f}[{\mathbf{x}}_i,\varphi ]{)}^2$$

$$\frac{\partial {\ell }_i}{\partial \text{f}[{\mathbf{x}}_i,\varphi ]}=2(y_i-\text{f}[{\mathbf{x}}_i,\varphi ])(-1)=2(\text{f}[{\mathbf{x}}_i,\varphi ]-y_i)$$

Problem 7.5

Calculate the derivative $\frac{\partial {\ell }_i}{\partial \text{f}[{\mathbf{x}}_i,\varphi ]}$ for the binary classification loss function:

$${\ell }_i=-(1-y_i)\log [1-\text{sig}[\text{f}[{\mathbf{x}}_i,\varphi ]\left]\right]-y_i\log \left[\text{sig}\right[\text{f}[{\mathbf{x}}_i,\varphi ]\left]\right]$$

For the sake of clean math let’s write $f=\text{f}[{\mathbf{x}}_i,\varphi ]$. So we want to find $\frac{\partial {\ell }_i}{\partial f}$.

We have:

$$\frac{\partial {\ell }_i}{\partial f}=\frac{\partial {\ell }_i}{\partial \text{sig}[f]}\frac{\partial \text{sig}[f]}{\partial f}$$

The first term is:

$$\begin{array}{rl}\frac{\partial {\ell }_i}{\partial \text{sig}[f]}& =-(1-y_i)\left(\frac{1}{1-\text{sig}[f]}\right)(-1)-(y_i)\left(\frac{1}{\text{sig}[f]}\right)(1)\\ & =\frac{1-y_i}{1-\text{sig}[f]}-\frac{y_i}{\text{sig}[f]}\end{array}$$

The second term is:

$$\frac{\partial \text{sig}[f]}{\partial f}=\frac{\exp [-f]}{(1+\exp [-f]{)}^2}$$

Combining them back, we have:

$$\frac{\partial {\ell }_i}{\partial f}=\left(\frac{1-y_i}{1-\text{sig}\left[f\right]}-\frac{y_i}{\text{sig}[f]}\right)\left(\frac{\exp [-f]}{(1+\exp [-f]{)}^2}\right)$$

Recall that the sigmoid is:

$$\begin{array}{rl}\text{sig}[f]& =\frac{1}{1+\exp [-f]}\\ 1-\text{sig}[f]& =\frac{\exp [-f]}{1+\exp [-f]}\end{array}$$

Let’s substitute these back into our loss function and simplify:

\begin{align} \frac{ \partial \ell_{i} }{ \partial f } & = \left( (1-y_{i}) \left( \frac{1+\exp[-f]}{\exp[-f]} \right) - (y_{i}) (1+\exp[-f]) \right) \left( \frac{\exp[-f]}{(1+\exp[-f])^{2}} \right) \\[2ex] & = \left( \frac{(1-y_{i})(1+\exp[-f])}{\exp[-f]} - (y_{i}) (1+\exp[-f])\right)\left( \frac{\exp[-f]}{(1+\exp[-f])^{2}} \right) \\[2ex] & =\cancel{ (1+\exp[-f]) }\left( \frac{1-y_{i}}{\exp[-f]} - y_{i} \right)\left( \frac{\exp[-f]}{(1+\exp[-f])^\cancel{ {2} }} \right)\\[2ex] & =\left( \frac{1-y_{i}}{\exp[-f]}-y_{i} \right)\left( \frac{\exp[-f]}{1+\exp[-f]} \right) \\[2ex] & = \frac{1-y_{i}}{1+\exp[-f]}-\frac{y_{i}\exp[-f]}{1+\exp[-f]} \\[2ex] & = \frac{1-y_{i}-y_{i}\exp[-f]}{1+\exp[-f]} \\[2ex] & = \frac{1}{1+\exp[-f]}- \frac{y_{i}+y_{i}\exp[-f]}{1+\exp[-f]} \\[2ex] & = \frac{1}{1+\exp[-f]}-\frac{y_{i}(1+\exp[-f])}{1+\exp[-f]} \\[2ex] & = \frac{1}{1+\exp[-f]} - y_{i} \\[2ex] & =\boxed{\text{sig}[f]-y_{i}} \end{align}

Nice result (and seems important)!

Problem 7.6

Show that for $\mathbf{z}=\beta +\mathbf{\Omega h}$:

$$\frac{\partial \mathbf{z}}{\partial \mathbf{h}}={\mathbf{\Omega }}^T$$

where $\partial \mathbf{z}/\partial \mathbf{h}$ is a matrix containing the term $\partial z_i/\partial h_j$ in its $i$-th column and $j$-th row.

To do this, first find an expression for the constituent elements $\partial z_i/\partial h_j$, and then consider the form that the matrix $\partial \mathbf{z}/\partial \mathbf{h}$ must take.

Let:

$$\mathbf{h}=\left[\begin{array}{c}{h}_1\\ ⋮\\ h_n\end{array}\right],\mathbf{z}=\beta +\Omega \mathbf{h},\mathbf{\Omega }=[{\Omega }_{ij}]\in {\mathbb{R}}^{m\times n}$$

so that $z_i={\beta }_i+{\sum }_{k=1}^n{\mathbf{\Omega }}_{ik}{h}_k$. We can see this is in action in Example Computation – ${\mathbf{\Omega }}_{ik}{\mathbf{h}}_k$ is a dot product between the $i$-th row of $\Omega$ and $\mathbf{h}$, and then we add the bias for that row.

Now let’s consider the element-wise derivative. For some fixed $i$ and $j$:

$$\frac{\partial z_i}{\partial h_j}=\frac{\partial }{\partial h_j}\left({\beta }_i+\sum _k{\mathbf{\Omega }}_{ik}{h}_k\right)={\mathbf{\Omega }}_{ij}$$

This is because $z_i$ depends linearly on each $h_j$, so the partial derivative just plucks out the corresponding weight.

Then, since ”$\partial \mathbf{z}/\partial \mathbf{h}$ is a matrix containing the term $\partial z_i/\partial h_j$ in its $i$-th column and $j$-th row”:

$$\frac{\partial \mathbf{z}}{\partial \mathbf{h}}={\left[\frac{\partial z_i}{\partial h_j}\right]}_{\text{row }j,\text{col }i}={\Omega }_{ij}$$

Now we can notice that the matrix whose $(j,i)$ entry is ${\Omega }_{ij}$ is precisely the transpose of $\mathbf{\Omega }$, leading us to:

$$\frac{\partial \mathbf{z}}{\partial \mathbf{h}}={\mathbf{\Omega }}^T$$

If you instead store instead in row $i$, column $j$ —the more common “Jacobian” convention—the matrix would be $\mathbf{\Omega }$ itself. The textbook’s row/column choice therefore introduces the transpose.

Problem 7.7

Consider the case where we use the logistic sigmoid as an activation function, so $h=\text{sig}[f]$. Compute the derivative $\partial h/\partial f$ for this activation function. What happens to the derivative when the input takes (i) a large positive value and (ii) a large negative value?

We have:

$$h=\text{sig}[f]=\frac{1}{1+\exp [-f]}$$

and it follows that

$$\frac{\partial h}{\partial f}=\frac{\partial \text{sig}[f]}{\partial f}=\frac{\exp [-f]}{(1+\exp [-f]{)}^2}$$

which can then be re-written as:

$$\begin{array}{rl}\frac{\partial h}{\partial f}& =\frac{1}{1+\exp [-f]}\cdot \frac{\exp [-f]}{1+\exp [-f]}\\ & =\text{sig}[f]\cdot (1-\text{sig}[f])\\ & =h(1-h)\end{array}$$

When the input $f$ becomes large positive value, $h=\text{sig}[f]\approx 1$, so we have $1(1-1)=0$. When the input $f$ becomes a large negative value, we in turn have $h\approx 0$, which gives $0(1-0)=0$. Thus, the gradient is (near) zero at both extremes.

Problem 7.8

Consider using (i) the Heaviside function and (ii) the rectangular function as activations:

$$\text{Heaviside}[z]=\{\begin{array}{ll}0& z<0\\ 1& z\ge 0\end{array}$$ $$\text{rect}[z]=\{\begin{array}{ll}0& z<0\\ 1& 0\le z\le 1\\ 0& z>1\end{array}$$

Discuss why these functions are problematic for neural network training with gradient-based optimization methods.

Both of these functions are flat and discontinuous.

In regions where the function is flat, weights before the activation will not change because the chain rule will include a multiplication by zero. For some weight $w$:

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial a}\cdot \frac{\partial a}{\partial z}\cdot \frac{\partial z}{\partial w}$$

In this case $\frac{\partial a}{\partial z}=0$ because the activation is flat, so we also just have $\frac{\partial L}{\partial w}=0$ and the weight doesn’t update at all.

In regions where the function is discontinuous, there’s no gradient to follow (undefined) – optimization algorithms like gradient descent are stuck.

Problem 7.9

Consider a loss function $\ell [\mathbf{f}]$, where $\mathbf{f}=\beta +\mathbf{\Omega h}$. We want to find how the loss $\ell$ changes when we change $\mathbf{\Omega }$, which we’ll express with a matrix that contains the derivative $\partial \ell /\partial {\Omega }_{ij}$ at the $i$-th row and $j$-th column. Find an expression for $\partial f_i/\partial {\Omega }_{ij}$, and, using the chain rule, show that

$$\frac{\partial \ell }{\partial \Omega }=\frac{\partial \ell }{\partial \mathbf{f}}{\mathbf{h}}^T$$

We have

$$f_i={\beta }_i+\sum _j{\mathbf{\Omega }}_{ij}{h}_j$$

and so:

$$\frac{\partial f_i}{\partial {\mathbf{\Omega }}_{ij}}=h_j$$

Using the chain rule:

$$\frac{\partial \ell }{\partial {\mathbf{\Omega }}_{ij}}=\frac{\partial \ell }{\partial f_i}\frac{\partial f_i}{\partial {\mathbf{\Omega }}_{ij}}=\frac{\partial \ell }{\partial f_i}{h}_j$$

Converting back to vector form, we have

$$\frac{\partial \ell }{\partial \mathbf{\Omega }}=\frac{\partial \ell }{\partial \mathbf{f}}{\mathbf{h}}^T$$

as required.

Problem 7.10

Derive the equations for the backward pass of the backpropagation algorithm that uses leaky ReLU activations, which are defined as:

$$\mathbf{a}[z]=\text{ReLU}[z]=\{\begin{array}{ll}\alpha \cdot z& z<0\\ z& z\ge 0\end{array}$$

where $\alpha$ is a small positive constant, typically $0.1$.

The Leaky ReLU has a gradient of $+1$ when the input is greater than zero, and a gradient of $\alpha$ when the gradient is less than zero. The backprop equations are the same except for

$$\frac{\partial {\ell }_i}{\partial {\mathbf{f}}_{k-1}}=\mathbb{I}[{\mathbf{f}}_{k-1}>0]\odot \left({\mathbf{\Omega }}_k^T\frac{\partial {\ell }_i}{\partial {\mathbf{f}}_{\mathbf{k}}}\right)+\mathbb{I}[{\mathbf{f}}_{\mathbf{k}-1}<0]\odot \alpha \left({\mathbf{\Omega }}_k^T\frac{\partial {\ell }_i}{\partial {\mathbf{f}}_k}\right)$$

Problem 7.11

Problem 7.12

Problem 7.13

Problem 7.14

Problem 7.15

Problem 7.16

Problem 7.17