UDL Chapter 3 Problems

Problem 3.1

What kind of mapping from input to output would be created if the activation function in equation

$$\begin{array}{rl}y& =f[x,\varphi ]\\ & ={\varphi }_0+{\varphi }_{1}a[{\theta }_{10}+{\theta }_{11}x]+{\varphi }_{2}a[{\theta }_{20}+{\theta }_{21}x]+{\varphi }_{3}a[{\theta }_{30}+{\theta }_{31}x]\end{array}$$

was linear so that $a[z]={\psi }_0+{\psi }_{1}z$? What kind of mapping would be created if the activation function was removed, so $a[z]=z$?

If each activation function was linear, the input-output mapping would also be a linear function. We would have

$$\begin{array}{rl}y=& {\varphi }_0+{\varphi }_1({\psi }_0+{\psi }_1{\theta }_{10}+{\psi }_1{\theta }_{11}x)\\ & +{\varphi }_2({\psi }_0+{\psi }_1{\theta }_{20}+{\psi }_1{\theta }_{21}x)\\ & +{\varphi }_3({\psi }_0+{\psi }_1{\theta }_{30}+{\psi }_1{\theta }_{31}x)\end{array}$$

Grouping terms gives us a linear function of $x$:

$$y=(\text{constant})+(\text{linear terms involving }x)$$

If the activation function was removed, it would also simply be a linear function.

If $a[z]=z$, we would simply have

$$\begin{array}{rl}y& ={\varphi }_0+{\varphi }_1({\theta }_{10}+{\theta }_{11}x)+{\varphi }_2({\theta }_{20}+{\theta }_{21}x)+{\varphi }_3({\theta }_{30}+{\theta }_{31}x)\\ & ={\varphi }_0+({\varphi }_1{\theta }_{10}+{\varphi }_2{\theta }_{20}+{\varphi }_3{\theta }_{30})+({\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21}+{\varphi }_3{\theta }_{31})x\end{array}$$

which is also a linear function of $x$.

Problem 3.2

For each of the four linear regions in figure 3.3j, indicate which hidden units are inactive and which are active (i.e., which do and do not clip their inputs).

• Region 1: $h_3$ is active
• Region 2: $h_1$ and $h_3$ are active
• Region 3: $h_1,h_2,h_3$ are all active
• Region 4: $h_1$ and $h_2$ are active

Problem 3.3

Derive expressions for the positions of the “joints” in function in figure 3.3j in terms of the ten parameters $\varphi$ and input $x$. Derive expressions for slopes of the four linear regions.

Joints are created by ReLU activations when a hidden unit has a joint at that point. Thus, we set each by the function computed before the ReLU is applied to zero.

• Joint 1: ${\theta }_{10}+{\theta }_{11}x=0⟶x=-\frac{{\theta }_{10}}{{\theta }_{11}}$
• Joint 2: ${\theta }_{20}+{\theta }_{21}x=0⟶x=-\frac{{\theta }_{20}}{{\theta }_{21}}$
• Joint 3: ${\theta }_{30}+{\theta }_{31}x=0⟶x=-\frac{{\theta }_{30}}{{\theta }_{31}}$

Slopes for each region are determined by which hidden units are active in that region.

• Region 1: $h_3$ is active, so we have $m_1={\varphi }_3{\theta }_{31}$
• Region 2: $h_1$ and $h_3$ are active, so we have $m_2=({\varphi }_1{\theta }_{11}+{\varphi }_3{\theta }_{31})x$
• Region 3: $h_1,h_2,h_3$ are all active, so we have $m_3=({\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21}+{\varphi }_3{\theta }_{31})x$
• Region 4: $h_1$ and $h_2$ are active, so we have $m_4=({\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21})x$

Problem 3.4

Draw a version of figure 3.3 where the $y$-intercept and slope of the 3rd hidden unit have changed as in 3.14c. Assume that the remaining parameters remain the same.

Problem 3.5

Prove that the following property holds for $\alpha \in {\mathbb{R}}^{+}$:

$$\text{ReLU}[\alpha \cdot z]=\alpha \cdot \text{ReLU}[z]$$

This is known as the non-negative homogeneity property of the ReLU function.

We have

$$\text{ReLU}[z]=\{\begin{array}{ll}0& z<0\\ z& z\ge 0\end{array}=\text{max}(0,z)$$

Thus,

$$\begin{array}{rl}\text{ReLU}[\alpha \cdot z]& =\text{max}(0,\alpha \cdot z)\\ & =\alpha \cdot \text{max}(0,z)\\ & =\alpha \cdot \text{ReLU}(z)\end{array}$$

Problem 3.6

Following on from problem 3.5, what happens to the shallow network defined in equations 3.3 and 3.4 when we multiply the parameters ${\theta }_{10}$ and ${\theta }_{11}$ by a positive constant $\alpha$ and divide the slope ${\varphi }_1$ by the same parameter $\alpha$? What happens if $\alpha$ is negative?

Equation 3.3 defines hidden units:

$$h_1=a[{\theta }_{10}+{\theta }_{11}x]$$

Equation 3.4 defines the network:

$$y={\varphi }_0+{\varphi }_1{h}_1,\dots$$

Multiplying $h_1$ by $\alpha$ and then dividing by it again (by way of ${\varphi }_1$) would have no effect:

$$\begin{array}{rl}y& ={\varphi }_0+\frac{{\varphi }_1}{\alpha }(\alpha a[{\theta }_{10}+{\theta }_{11}x])\\ & ={\varphi }_0+{\varphi }_1(a[{\theta }_{10}+{\theta }_{11x}])\end{array}$$

Problem 3.7

Consider fitting the model in equation 3.1 using a least squares loss function. Does this loss function have a unique minimum? i.e., is there a single “best” set of parameters.

The least squares loss function for this model does not necessarily have a unique minimum, because the non-linearities introduced by the activation function can create a non-convex optimization landscape. Furthermore, different parameter combinations to produce the same predictions, leading to equivalent solutions.

Problem 3.8

Consider replacing the ReLU activation function with (i) the Heaviside step function $\text{heaviside}(z)$, (ii) the hyperbolic tangent function $\tanh [z]$, and (iii) the rectangular function $\text{rect}[z]$, where

$$\text{heaviside}[z]=\{\begin{array}{ll}0& z<0\\ 1& z\ge 0\end{array}$$

and

$$\text{rect}[z]=\{\begin{array}{ll}0& z<0\\ 1& 0\le z\le 1\\ 0& z>1\end{array}$$

Redraw a version of figure 3.3 for each of these functions. The original parameters were: $\varphi =\{{\varphi }_0,{\varphi }_1,{\varphi }_2,{\varphi }_3,{\theta }_{10},{\theta }_{11},{\theta }_{20},{\theta }_{21},{\theta }_{30},{\theta }_{31}\}=\{-0.23,-1.3,1.3,0.66,-0.2,0.4,-0.9,0.9,1.1,-0.7\}.$ Provide an informal description of the family of functions that can be created by neural networks with one input, three hidden units, and one output for each activation function.

Heaviside: We would produce piecewise constant functions with up to four linear regions. Each hidden unit creates a single step in the output at its $x$-intercept. The output weights and biases of the network can adjust the heights of these constant segments, but the general shape remains a sequence of horizontal steps.

Hyperbolic tangent: This network would produce smooth piecewise non-linear functions. These functions can approximate continuous, differentiable curves, with complexity determined by the arrangement and blending of the sigmoids created by the hidden units.

Rectangular: This network would produce piecewise constant functions with rectangular pulses. The functions can have up to three non-overlapping or overlapping “bumps,” depending on how the hidden units’ activations interact.

Problem 3.9

Show that the third linear region in figure 3.3 has a slope that is the sum of the slopes of the first and fourth linear regions

From problem 3.3, we had:

• Region 1: $h_3$ is active, so we have $m_1={\varphi }_3{\theta }_{31}$
• Region 3: $h_1,h_2,h_3$ are all active, so we have $m_3={\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21}+{\varphi }_3{\theta }_{31}$
• Region 4: $h_1$ and $h_2$ are active, so we have $m_4={\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21}$

Thus, it is to show that:

$$\begin{array}{rl}{m}_3& =m_1+m_4\\ {\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21}+{\varphi }_3{\theta }_{31}& =({\varphi }_3{\theta }_{31})+({\varphi }_1{\theta }_{11}+{\varphi }_2{\theta }_{21})\end{array}$$

Problem 3.10

Consider a neural network with one input, one output, and three hidden units. The construction in figure 3.3 shows how this creates four linear regions. Under what circumstances could this network produce a function with fewer than four linear regions?

The four linear regions result from the 3 hidden units creating 3 joints from the non-linearities caused by the ReLU functions. We can have less joints if they occur at the same place, i.e. two or more of the hidden units have the same $x$-intercept. This could result from one or more hidden units remaining inactive for all relevant inputs, or if the weights and biases of the hidden units are proportional to each other.

Problem 3.11

How many parameters does the model in figure 3.6 have?

Each of the four hidden unit has 1 weight and 1 bias. Each of the two output unit has four weights and 1 bias. In total:

$$(4\times (1+1))+(2\times (4+1))=8+10=18$$

Problem 3.12

How many parameters does the model in figure 3.7 have?

Each of the 3 hidden units has 2 weights and 1 bias (see Equation 3.9). The output unit has 3 weights and 1 bias:

$$(3\times (2+1))+(3+1)=9+4=13$$

Problem 3.13

What is the activation pattern for each of the seven regions in figure 3.8? In other words, which hidden units are active (pass the input) and which are inactive (clip the input) for each region?

• Region 1: $h_1$
• Region 2: $h_1,h_2$
• Region 3: None
• Region 4: $h_1,h_3$
• Region 5: $h_1,h_2,h_3$
• Region 6: $h_3$
• Region 7: $h_2,h_3$

Problem 3.14

Write out the equations that define the network in figure 3.11. There should be three equations to compute the three hidden units from the inputs and two equations to compute the outputs from the hidden units.

Hidden units:

$$\begin{array}{rl}{h}_1& =a[{\theta }_{10}+{\theta }_{11}{x}_1+{\theta }_{12}{x}_2+{\theta }_{13}{x}_3]\\ h_2& =a[{\theta }_{20}+{\theta }_{21}{x}_1+{\theta }_{22}{x}_2+{\theta }_{23}{x}_3]\\ h_3& =a[{\theta }_{30}+{\theta }_{31}{x}_1+{\theta }_{32}{x}_2+{\theta }_{33}{x}_3]\end{array}$$

Outputs:

$$\begin{array}{rl}{y}_1& ={\varphi }_{10}+{\varphi }_{11}{h}_1+{\varphi }_{12}{h}_2+{\varphi }_{13}{h}_3\\ y_2& ={\varphi }_{20}+{\varphi }_{21}{h}_1+{\varphi }_{22}{h}_2+{\varphi }_{23}{h}_3\end{array}$$

Problem 3.15

What is the maximum possible number of 3D linear regions that can be created by the network in figure 3.11?

Each hidden unit divides the input space into two regions: one where the linear function inside $a$ is positive, and one where it is zero. The maximum number of regions created by 3 planes intersecting in 3D space, following figure 3.10c, would be 8 regions.

Can also use Zaslavsky’s result (see problem 3.18) to get:

$$\begin{array}{rl}R& =\sum _{k=0}^3\left(\genfrac{}{}{0ex}{}{3}{k}\right)=\left(\genfrac{}{}{0ex}{}{3}{0}\right)+\left(\genfrac{}{}{0ex}{}{3}{1}\right)+\left(\genfrac{}{}{0ex}{}{3}{2}\right)+\left(\genfrac{}{}{0ex}{}{3}{3}\right)\\ & =1+3+3+1=8\end{array}$$

Problem 3.16

Write out the equations for a network with two inputs, four hidden units, and three outputs.

• Two inputs: $x_1,x_2$
• Four hidden units:

$$\begin{array}{r}{h}_1={\theta }_{10}+{\theta }_{11}{x}_1+{\theta }_{12}{x}_2\\ h_2={\theta }_{20}+{\theta }_{21}{x}_1+{\theta }_{22}{x}_2\\ h_3={\theta }_{30}+{\theta }_{31}{x}_1+{\theta }_{32}{x}_2\\ h_4={\theta }_{40}+{\theta }_{41}{x}_1+{\theta }_{42}{x}_2\end{array}$$

• Three outputs:

$$\begin{array}{r}{y}_1={\varphi }_{10}+{\varphi }_{11}{h}_1+{\varphi }_{12}{h}_2+{\varphi }_{13}{h}_3+{\varphi }_{14}{h}_4\\ y_2={\varphi }_{20}+{\varphi }_{21}{h}_1+{\varphi }_{22}{h}_2+{\varphi }_{23}{h}_3+{\varphi }_{24}{h}_4\\ y_3={\varphi }_{30}+{\varphi }_{31}{h}_1+{\varphi }_{32}{h}_2+{\varphi }_{33}{h}_3+{\varphi }_{34}{h}_4\end{array}$$

Problem 3.17

Equations 3.11 and 3.12 define a general neural network with $D_i$ inputs, one hidden layer containing $D$ hidden units, and $D_o$ outputs. Find an expression for the number of parameters in the model in terms of $D_i$, $D$, and $D_o$.

$$((D_i+1)\times D)+((D+1)\times D_o)$$

See 3.11 and 3.12 for examples.

Problem 3.18

Show that the maximum number of regions created by a shallow network with $D_i=2$-dimensional input, $D_o=1$-dimensional output, and $D=3$ hidden units is seven, as in figure 3.8j. Use the result of Zaslavsky (1975) that the maximum number of regions created by partitioning a $D_i$-dimensional space with $D$ hyperplanes is ${\sum }_{j=0}^{D_i}\left(\genfrac{}{}{0ex}{}{D}{j}\right)$. What is the maximum number of regions if we add two more hidden units to this model, so $D=5$?

Original:

$$N=\sum _{j=0}^{D_i}\left(\genfrac{}{}{0ex}{}{D}{j}\right)=\left(\genfrac{}{}{0ex}{}{3}{0}\right)+\left(\genfrac{}{}{0ex}{}{3}{1}\right)+\left(\genfrac{}{}{0ex}{}{3}{2}\right)=1+3+3=7$$

With $D=5$:

$$N=\sum _{j=0}^{D_i}\left(\genfrac{}{}{0ex}{}{D}{j}\right)=\left(\genfrac{}{}{0ex}{}{5}{0}\right)+\left(\genfrac{}{}{0ex}{}{5}{1}\right)+\left(\genfrac{}{}{0ex}{}{5}{2}\right)=1+5+10=16$$