UDL Chapter 7 Problems

Problem 7.1

A two-layer network with two hidden units in each layer can be defined as:

$$\begin{array}{rl}y={\varphi }_0& +{\varphi }_{1}a[{\psi }_{01}+{\psi }_{11}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{21}a[{\theta }_{02}+{\theta }_{12}x]]\\ & +{\varphi }_{2}a[{\psi }_{02}+{\psi }_{12}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{22}a[{\theta }_{02}+{\theta }_{12}x]]\end{array}$$

where the functions $a[\bullet ]$ are ReLU functions. Compute the derivatives of the output $y$ with respect to each of the 13 parameters ${\varphi }_{\bullet },{\theta }_{\bullet \bullet }$ and ${\psi }_{\bullet \bullet }$ directly. The derivatives of the ReLU function with respect to its input $\partial a[z]/\partial [z]$ is the indicator function $\mathbb{I}[z>0]$, which returns one if the argument is greater than zero and zero otherwise.

Output layer:

$$\begin{array}{rl}\frac{\partial y}{\partial {\varphi }_0}& =1\\ \frac{\partial y}{\partial {\varphi }_1}& =a[{\psi }_{01}+{\psi }_{11}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{21}a[{\theta }_{02}+{\theta }_{12}x]]\\ \frac{\partial y}{\partial {\varphi }_2}& =a[{\psi }_{02}+{\psi }_{12}a[{\theta }_{01}+{\theta }_{11}x]+{\psi }_{22}a[{\theta }_{02}+{\theta }_{12}x]]\end{array}$$

Second hidden layer: Let us first define

$$\begin{array}{r}{h}_{21}={\psi }_{01}+{\psi }_{11}\cdot a[z_1]+{\psi }_{21}\cdot a[z_2]\\ h_{22}={\psi }_{02}+{\psi }_{12}\cdot a[z_1]+{\psi }_{22}\cdot a[z_2]\end{array}$$

Then:

$$\begin{array}{rl}\frac{\partial y}{\partial {\psi }_{01}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{01}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot 1\\ \frac{\partial y}{\partial {\psi }_{11}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{11}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot a[{\theta }_{01}+{\theta }_{11}x]\\ \frac{\partial y}{\partial {\psi }_{21}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial {\psi }_{21}}{\partial h_{21}}={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot a[{\theta }_{02}+{\theta }_{12}x]\\ \frac{\partial y}{\partial {\psi }_{02}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{02}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot 1\\ \frac{\partial y}{\partial {\psi }_{12}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{12}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot a[{\theta }_{01}+{\theta }_{11}x]\\ \frac{\partial y}{\partial {\psi }_{22}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial {\psi }_{22}}{\partial h_{22}}={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot a[{\theta }_{02}+{\theta }_{12}x]\end{array}$$

First hidden layer: Let us first define

$$\begin{array}{r}{h}_{11}={\theta }_{01}+{\theta }_{11}x\\ h_{12}={\theta }_{02}+{\theta }_{12}x\end{array}$$

Then we have:

$$\begin{array}{rl}\frac{\partial y}{\partial {\theta }_{01}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial h_{21}}{\partial a[h_{11}]}\frac{\partial a[h_{11}]}{\partial h_{11}}\frac{\partial h_{11}}{\partial {\theta }_{01}}\\ & ={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot {\psi }_{11}\cdot \mathbb{I}(h_{11}>0)\cdot 1\\ \frac{\partial y}{\partial {\theta }_{11}}& =\frac{\partial y}{\partial a[h_{21}]}\frac{\partial a[h_{21}]}{\partial h_{21}}\frac{\partial h_{21}}{\partial a[h_{11}]}\frac{\partial a[h_{11}]}{\partial h_{11}}\frac{\partial h_{11}}{\partial {\theta }_{11}}\\ & ={\varphi }_1\cdot \mathbb{I}(h_{21}>0)\cdot {\psi }_{11}\cdot \mathbb{I}(h_{11}>0)\cdot x\\ \frac{\partial y}{\partial {\theta }_{02}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial h_{22}}{\partial a[h_{12}]}\frac{\partial a[h_{12}]}{\partial h_{12}}\frac{\partial h_{12}}{\partial {\theta }_{02}}\\ & ={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot {\psi }_{12}\cdot \mathbb{I}(h_{12}>0)\cdot 1\\ \frac{\partial y}{\partial {\theta }_{12}}& =\frac{\partial y}{\partial a[h_{22}]}\frac{\partial a[h_{22}]}{\partial h_{22}}\frac{\partial h_{22}}{\partial a[h_{12}]}\frac{\partial a[h_{12}]}{\partial h_{12}}\frac{\partial h_{12}}{\partial {\theta }_{12}}\\ & ={\varphi }_2\cdot \mathbb{I}(h_{22}>0)\cdot {\psi }_{12}\cdot \mathbb{I}(h_{12}>0)\cdot x\end{array}$$

Problem 7.2

Problem 7.3

Problem 7.4

Problem 7.5

Problem 7.6

Problem 7.7

Problem 7.8

Problem 7.9

Problem 7.10

Problem 7.11

Problem 7.12

Problem 7.13

Problem 7.14

Problem 7.15

Problem 7.16

Problem 7.17