Composing Shallow Networks

As an entrypoint to deep neural networks, we first consider composing two shallow neural networks so that the output of the first becomes the input of the second.

Consider two shallow networks with 3 hidden units each.

The first network takes an input $x$ and returns output $y$ and is defined by:

$$\begin{array}{r}{h}_1=a[{\theta }_{10}+{\theta }_{11}x]\\ h_2=a[{\theta }_{20}+{\theta }_{21}x]\\ h_3=a[{\theta }_{30}+{\theta }_{31}x]\end{array}$$

and

$$y={\varphi }_0+{\varphi }_1{h}_1+{\varphi }_2{h}_2+{\varphi }_3{h}_3$$

The second network takes $y$ as input and returns $y^{\prime }$ and is defined by:

$$\begin{array}{r}{h}_1^{\prime }=a[{\theta }_{10}^{\prime }+{\theta }_{11}^{\prime }y]\\ h_2^{\prime }=a[{\theta }_{20}^{\prime }+{\theta }_{21}^{\prime }y]\\ h_3^{\prime }=a[{\theta }_{30}^{\prime }+{\theta }_{31}^{\prime }y]\end{array}$$

and

$$y^{\prime }={\varphi }_0^{\prime }+{\varphi }_1^{\prime }{h}_1^{\prime }+{\varphi }_2^{\prime }{h}_2^{\prime }+{\varphi }_3^{\prime }{h}_3^{\prime }$$

• The first network maps inputs $x\in [-1,1]$ to outputs $y\in [-1,1]$ using a function comprising three linear regions that are chosen so that they alternate the sign of their slope (4th linear region is outside range of graph). Multiple inputs $x$ (gray circles) now map to the same output $y$ (cyan circle).
• The second network defines a function comprising three linear regions that takes $y$ and returns $y^{\prime }$ (i.e. the cyan circle is mapped to the brown circle).
• The combined effect of these 2 functions when composed is that three different inputs $x$ are mapped to any given value of $y$ by the first network and are processed the same way by the second network.
• The result is that the function defined by the second network in panel (c) is duplicated three times, variously flipped and rescaled according to the slope of the regions of panel (b).

With ReLU activations, this model also describes a family of piecewise linear functions. However, the number of linear regions is potentially greater than for a shallow network with 6 hidden units. To see this, consider choosing the first network to produce three alternating regions of positive and negative slope (panel b above). This means that three different ranges of $x$ are mapped to the same output range $y\in [-1,1]$, and the subsequent mapping from this range of $y$ to $y^{\prime }$ is applied three times. The overall effect is that the function defined by the second network is duplicated three times to create nine linear regions.

The same principle applies in higher dimensions as well.

A different way to think about composing networks is that the first network “folds” input space $x$ onto itself so that multiple inputs generate the same output. Then the second network applies a function, which is replicated at all points that were folded on top of one another.

From composing networks to deep networks

We can show that our composition of networks is a special case of a Deep Neural Network with two hidden layers.

The output of the first network ($y={\varphi }_0+{\varphi }_1{h}_1+{\varphi }_2{h}_2+{\varphi }_3{h}_3$) is a linear combination of the activations of the hidden units. The first operations of the second network (${\theta }_{10}^{\prime }+{\theta }_{11}^{\prime }y,{\theta }_{20}^{\prime }+{\theta }_{21}^{\prime }y,{\theta }_{30}^{\prime }+{\theta }_{31}^{\prime }y$) are linear in the output of the first network. Applying one linear function to another yields another linear function.

Substituting the expression for $y$ in the calculation of hidden units in the second network gives:

$$\begin{array}{r}{h}_1^{\prime }=a[{\theta }_{10}^{\prime }+{\theta }_{11}^{\prime }y]=a[{\theta }_{10}^{\prime }+{\theta }_{11}^{\prime }{\varphi }_0+{\theta }_{11}^{\prime }{\varphi }_1{h}_1+{\theta }_{11}^{\prime }{\varphi }_2{h}_2+{\theta }_{11}^{\prime }{\varphi }_3{h}_3]\\ h_2^{\prime }=a[{\theta }_{20}^{\prime }+{\theta }_{21}^{\prime }y]=a[{\theta }_{20}^{\prime }+{\theta }_{21}^{\prime }{\varphi }_0+{\theta }_{21}^{\prime }{\varphi }_1{h}_1+{\theta }_{21}^{\prime }{\varphi }_2{h}_2+{\theta }_{21}^{\prime }{\varphi }_3{h}_3]\\ h_3^{\prime }=a[{\theta }_{30}^{\prime }+{\theta }_{31}^{\prime }y]=a[{\theta }_{30}^{\prime }+{\theta }_{31}^{\prime }{\varphi }_0+{\theta }_{31}^{\prime }{\varphi }_1{h}_1+{\theta }_{31}^{\prime }{\varphi }_2{h}_2+{\theta }_{31}^{\prime }{\varphi }_3{h}_3]\end{array}$$

which can be re-written as:

$$\begin{array}{r}{h}_1^{\prime }=a[{\psi }_{10}+{\psi }_{11}{h}_1+{\psi }_{12}{h}_2+{\psi }_{13}{h}_3]\\ h_2^{\prime }=a[{\psi }_{20}+{\psi }_{21}{h}_1+{\psi }_{22}{h}_2+{\psi }_{23}{h}_3]\\ h_3^{\prime }=a[{\psi }_{30}+{\psi }_{31}{h}_1+{\psi }_{32}{h}_2+{\psi }_{33}{h}_3]\end{array}$$

where ${\psi }_{10}={\theta }_{10}^{\prime }+{\theta }_{11}^{\prime }{\varphi }_0,{\psi }_{11}={\theta }_{11}^{\prime }{\varphi }_1,{\psi }_{12}={\theta }_{11}^{\prime }{\varphi }_2$ and so on. The result is a network with two hidden layers.

It follows that a network with two layers can represent the family of functions created by passing the output of one single-layer network into another. It represents a broader family, because the nine slope parameters, ${\psi }_{11},{\psi }_{21},\dots ,{\psi }_{33}$ can take arbitrary values, whereas the original parameters are constrained to be the outer product $[{\theta }_{11}^{\prime },{\theta }_{21}^{\prime },{\theta }_{31}^{\prime }{]}^T[{\varphi }_1,{\varphi }_2,{\varphi }_3]$.