Mathematical Formulation of Test Error

Let’s make the notions of noise, bias, and variance mathematically precise.

Consider a 1D regression problem where the data generation has additive noise with variance ${\sigma }^2$. We can observe different outputs $y$ for the same input $x$; so for each $x$, there is a distribution $\text{Pr}(y|x)$ with expected value (mean) $\mu (x)$:

$$\mu [x]={\mathbb{E}}_y[y[x]]=\int y[x]\text{Pr}(y|x)dy$$

and fixed noise ${\sigma }^2={\mathbb{E}}_y[(\mu [x]-y[x]{)}^2]$.

• Here we have used the notation $y[x]$ to specify that we are considering the output $y$ at a given input position $x$.

Definition: Expectation

Consider a function $f(x)$ and a probability function $Pr(x)$ defined over $x$. The expected value of a function $f[\bullet ]$ of a random variable $x$ with respect to the probability $Pr(x)$ is defined as:

$${\mathbb{E}}_x[f[x]]=\int f[x]Pr(x)dx$$

Now consider a least squares loss between the model prediction $\text{f}[x,\varphi ]$ at position $x$ and the observed value $y[x]$ at that position:

$$\begin{array}{rl}L[x]& =(\text{f}[x,\varphi ]-y[x]{)}^2\\ & =((\text{f}[x,\varphi ]-\mu [x])+(\mu [x]-y[x]){)}^2\\ & =(\text{f}[x,\varphi ]-\mu [x]{)}^2+2(\text{f}[x,\varphi ]-\mu [x])(\mu [x]-y[x])+(\mu [x]-y[x]{)}^2\end{array}$$

where we have both added and subtracted the mean $\mu [x]$ of the underlying function in the second line and have expanded out the squared term in the third line.

The underlying function is stochastic, so this loss depends on the particular $y[x]$ we observe. The expected loss is:

$$\begin{array}{rl}{\mathbb{E}}_y[L[x]]& ={\mathbb{E}}_y[(\text{f}[x,\varphi ]-\mu [x]{)}^2+2(\text{f}[x,\varphi ]-\mu [x])(\mu [x]-y[x])+(\mu [x]-y[x]{)}^2]\\ & =(\text{f}[x,\varphi ]-\mu [x]{)}^2+2(\text{f}[x,\varphi ]-\mu [x])(\mu [x]-{\mathbb{E}}_y[y[x]])+{\mathbb{E}}_y[(\mu [x]-y[x]{)}^2]\\ & =(\text{f}[x,\varphi ]-\mu [x]{)}^2+2(\text{f}[x,\varphi ]-\mu [x])\cdot 0+{\mathbb{E}}_y[(\mu [x]-y[x]{)}^2]\\ & =(\text{f}[x,\varphi ]-\mu [x]{)}^2+{\sigma }^2\end{array}$$

where we have made use of the rules for manipulating expectations.

• In the second line, we have distributed the expectation operator and removed it from terms terms with no dependence on $y[x]$
• In the third line, we note that the second term is zero since ${\mathbb{E}}_y[y[x]]=\mu [x]$ by definition.
• In the fourth line, we have substituted in the definition of the noise ${\sigma }^2$.

We can see that the expected loss has been broken down into two terms: the first term is the squared deviation between the model and the true function mean, and the second term is the noise.

The first term can be further partitioned into bias and variance.

The parameters $\varphi$ of the model $\text{f}[x,\varphi ]$ depend on the training dataset $\mathcal{D}=\{x_i,y_i\}$; so to be more proper, we should write $\text{f}[x,\varphi [\mathcal{D}]]$. The training dataset is a random sample from the data generation process; with a different sample of training data, we would learn different parameter values. The expected model output ${\text{f}}_{\mu }[x]$ with respect to all datasets $\mathcal{D}$ is hence:

$${\text{f}}_{\mu }[x]={\mathbb{E}}_{\mathcal{D}}\left[\text{f}\right[x,\varphi [\mathcal{D}]\left]\right]$$

Returning to the first term of equation our equation, we add and subtract ${\text{f}}_{\mu }[x]$ and expand:

$$\begin{array}{rl}& (\text{f}[x,\varphi ]-\mu [x]{)}^2\\ & =(\text{f}[x,\varphi [\mathcal{D}]]-\mu [x]{)}^2\\ & =((\text{f}[x,\varphi [\mathcal{D}]]-f_{\mu }[x])+({\text{f}}_{\mu }[x]-\mu [x]){)}^2\\ & =((\text{f}[x,\varphi [\mathcal{D}]]-{\text{f}}_{\mu }[x]{)}^2+2(\text{f}[x,\varphi [\mathcal{D}]]-{\text{f}}_{\mu }[x])({\text{f}}_{\mu }[x]-\mu [x]))+({\text{f}}_{\mu }[x]-\mu [x{]}^2)\end{array}$$

We then take the expectation with respect to the training dataset $\mathcal{D}$:

$${\mathbb{E}}_D[(\text{f}[x,\varphi [\mathcal{D}]]-\mu [x]{)}^2]={\mathbb{E}}_D[(\text{f}[x,\varphi [\mathcal{D}]]-{\text{f}}_{\mu }[x]{)}^2]+({\text{f}}_{\mu }[x]-\mu [x]{)}^2$$

• We removed the middle term since it doesn’t depend on $\mathcal{D}$.

Finally, substituting this result gives:

$${\mathbb{E}}_D[{\mathbb{E}}_y[L[x]]]=\underset{\text{variance}}{\underbrace{{\mathbb{E}}_D[(\text{f}[x,\varphi [\mathcal{D}]]-{\text{f}}_{\mu }[x]{)}^2]}}+\underset{\text{bias}}{\underbrace{({\text{f}}_{\mu }[x]-\mu [x]{)}^2}}+\underset{\text{noise}}{\underbrace{{\sigma }^2}}$$

This equation says that the expected loss after considering the uncertainty in the training data $\mathcal{D}$ and the test data $y$ consists of three additive components.

• The variance is the uncertainty in the fitted model due to the particular training dataset we sample.
• The bias is the systematic deviation of the model from the mean of the function we are modeling.
• The noise is the inherently uncertainty in the true mapping from input to output.

These sources of error will be present for any task. We’ve seen that they combine additively for regression with least squares loss; however, their interaction can be more complex for other problem types.