UDL Chapter 4 Problems

Problem 4.1

Consider composing the two neural networks in figure 4.8. Draw a plot of the relationship between the input $x$ and output $y^{\prime }$ for $x\in [-1,1]$.

Problem 4.2

Identify the four parameters in figure 4.6.

• $K=3$
• $D_1=4$
• $D_2=2$
• $D_3=3$

Problem 4.3

Using the non-negative homogeneity property of the ReLU function (see problem 3.5), show that:

$$\text{ReLU}[{\beta }_1+{\lambda }_1\cdot {\Omega }_1\text{ReLU}[{\beta }_0+{\lambda }_0\cdot {\Omega }_0\mathbf{x}]]={\lambda }_0{\lambda }_1\cdot \text{ReLU}\left[\frac{1}{{\lambda }_0{\lambda }_1}{\beta }_1+{\Omega }_1\text{ReLU}\left[\frac{1}{{\lambda }_0}{\beta }_0+{\Omega }_0\mathbf{x}\right]\right]$$

where ${\lambda }_0$ and ${\lambda }_1$ are non-negative scalars. From this, we see that the weight matrices can be rescaled by any magnitude as long as the biases are also adjusted, and the scale factors can be re-applied at the end of the network.

The non-negative homogeneity property states that:

$$\text{ReLU}[\alpha \cdot z]=\alpha \cdot \text{ReLU}[z]$$

We have

$$\begin{array}{rl}& \text{ReLU}[{\beta }_1+{\lambda }_1\cdot {\Omega }_1\text{ReLU}[{\beta }_0+{\lambda }_0\cdot {\Omega }_0\mathbf{x}]]\\ & =\text{ReLU}\left[{\lambda }_1\cdot \left(\frac{{\beta }_1}{{\lambda }_1}+{\Omega }_1\text{ReLU}\left[{\lambda }_0\cdot \left(\frac{{\beta }_0}{{\lambda }_0}+{\Omega }_0\mathbf{x}\right)\right]\right)\right]\\ & ={\lambda }_1\text{ReLU}\left[\frac{{\beta }_1}{{\lambda }_1}+{\lambda }_0{\Omega }_1\text{ReLU}\left[\frac{{\beta }_0}{{\lambda }_0}+{\Omega }_0\mathbf{x}\right]\right]\\ & ={\lambda }_1\text{ReLU}\left[{\lambda }_0\left(\frac{{\beta }_1}{{\lambda }_1{\lambda }_0}+{\Omega }_1\text{ReLU}\left[\frac{{\beta }_0}{{\lambda }_0}+{\Omega }_0\mathbf{x}\right]\right)\right]\\ & ={\lambda }_0{\lambda }_1\text{ReLU}\left[\frac{{\beta }_1}{{\lambda }_1{\lambda }_0}+{\Omega }_1\text{ReLU}\left[\frac{{\beta }_0}{{\lambda }_0}+{\Omega }_0\mathbf{x}\right]\right]\end{array}$$

as desired.

Problem 4.4

Write out the equations for a deep neural network that takes $D_i=5$ inputs, $D_o=4$ outputs and has three hidden layers of sizes $D_1=20$, $D_2=10$, and $D_3=7$, respectively, in both the forms of equations 4.15 and 4.16. What are the sizes of each weight matrix ${\Omega }_{\bullet }$ and bias vector ${\beta }_{\bullet }$?

Individual equations (like 4.15):

$$\begin{array}{rl}{\mathbf{h}}_1& =a[{\beta }_0+{\Omega }_0\mathbf{x}]\\ {\mathbf{h}}_2& =a[{\beta }_1+{\Omega }_1{\mathbf{h}}_1]\\ {\mathbf{h}}_3& =a[{\beta }_2+{\Omega }_2{\mathbf{h}}_2]\\ \mathbf{y}& ={\beta }_3+{\Omega }_3{\mathbf{h}}_3\end{array}$$

One equation (like 4.16):

$$\mathbf{y}={\beta }_3+{\Omega }_{3}a[{\beta }_2+{\Omega }_{2}a[{\beta }_1+{\Omega }_{1}a[{\beta }_0+{\Omega }_0\mathbf{x}]]]$$

Sizes:

• ${\Omega }_0:20\times 5$
• ${\beta }_0:20$
• ${\Omega }_1:10\times 20$
• ${\beta }_1:10$
• ${\Omega }_2:7\times 10$
• ${\beta }_2:7$
• ${\Omega }_3:4\times 7$
• ${\beta }_3:4$

Problem 4.5

Consider a deep neural network with $D_i=5$ inputs, $D_o=1$ output, and $K=20$ hidden layers containing $D=30$ hidden units each. What is the depth of this network? What is the width?

• Depth is 20 (number of hidden layers)
  • Or 21 if we count the output layer? In this case, the definition we’re using is “number of layers with parameters” instead of number of hidden layers
• Width is 30 (number of hidden units in each layer)

Problem 4.6

Consider a network with $D_i=1$ input, $D_o=1$ output, and $K=10$ layers, with $D=10$ hidden units in each. Would the number of weights increase more if we increased the depth by one or the width by one?

Original:

• Input to $h_1$: $10\times 1=10$
• Between 9 hidden layers: $9\times 10\times 10=900$
• Last hidden to output: $10\times 1=10$
• Each hidden layer has 10 biases: $9\times 10=90$
• Output layer: $1$ bias
• Total: $10+900+10+90+1=1011$

Increase depth by 1:

• Input to $h_1$: $10\times 1=10$
• Between 9 hidden layers: $10\times 10\times 10=1000$
• Last hidden to output: $10\times 1=10$
• Each hidden layer has 10 biases: $10\times 10=100$
• Output layer: $1$ bias
• Total: $10+1000+10+100+1=1121$

Increase width by 1:

• Input to $h_1$: $11\times 1=11$
• Between 9 hidden layers: $9\times 11\times 11=1089$
• Last hidden to output: $11\times 1=11$
• Each hidden layer has 10 biases: $9\times 11=99$
• Output layer: $1$ bias
• Total: $11+1089+11+99+1=1211$

Problem 4.7

Choose values for the parameters $\varphi =\{{\varphi }_0,{\varphi }_1,{\varphi }_2,{\varphi }_3,{\theta }_{10},{\theta }_{11},{\theta }_{20},{\theta }_{21},{\theta }_{30},{\theta }_{31}\}$ for the shallow neural network in equation 3.1 (with ReLU activation functions) that will define an identity function over a finite range $x\in [a,b]$.

The function is:

$$\begin{array}{rl}y& =f[x,\varphi ]\\ & ={\varphi }_0+{\varphi }_{1}a[{\theta }_{10}+{\theta }_{11}x]+{\varphi }_{2}a[{\theta }_{20}+{\theta }_{21}x]+{\varphi }_{3}a[{\theta }_{30}+{\theta }_{31}x]\end{array}$$

We want it to be the identity function, such that

$$\begin{array}{rl}y& =f[x,\varphi ]\\ & =x\end{array}$$

If we have:

• ${\varphi }_0=0$
• ${\varphi }_1=1$
• ${\theta }_{10}=0$
• ${\theta }_{11}=1$
• ${\varphi }_2=1$
• ${\theta }_{20}=0$
• ${\theta }_{21}=1$
• ${\varphi }_3=1$
• ${\theta }_{30}=0$
• ${\theta }_{31}=1$

We would get $y=3\cdot \text{ReLU}[x]$, which only equals $x>0$ and has slope $3$ instead of $1$.

Instead, we need

• ${\varphi }_0=a$
• ${\varphi }_1=1$
• ${\theta }_{10}=-a$
• ${\theta }_{11}=1$
• ${\varphi }_2=-1$
• ${\theta }_{20}=-b$
• ${\theta }_{21}=1$
• ${\varphi }_3=0$
• ${\theta }_{30}=0$
• ${\theta }_{31}=1$

which expands to

$$\begin{array}{rl}f[x,\varphi ]& =a+\text{ReLU}[x-a]-\text{ReLU}[x-b]\\ & =a+\text{max}[0,x-a]-\text{max}[0,x-b]\end{array}$$

If we have:

• $x\le a$, both ReLUs are 0 so $f[x]=a$
• $a\le x\le b$, the first ReLU is active, so we have $f[x]=a+(x-a)=x$
• $x\ge b$, both ReLUs are active, so we have $f[x]=a+(x-a)-(x-b)=b$

Problem 4.8

Problem 4.9

Problem 4.10

Problem 4.11