Supervised Learning

A supervised learning model defines a mapping from one or more inputs to one or more outputs. The model is a mathematical equation; when the inputs are passed through tis equation, it computes the output (inference). The model equation also contains parameters. Different parameter values change the outcome of the computation; the model equation describes a family of possible input-output mappings, and the parameters specify the particular relationship.

When we train or learn a model, we find parameters that describe the true relationship between inputs and outputs. A learning algorithm takes a training set of input/output pairs and manipulates the parameters until the inputs predict their corresponding outputs as closely as possible.

Specifically, we aim to build a model that takes an input $\mathbf{x}$ and outputs a prediction $\mathbf{y}$, both of which are vectors. To make the prediction, we need a model $\mathbf{f}[\cdot ]$ that takes the input $\mathbf{x}$ and returns $\mathbf{y}$, so:

$$\mathbf{y}=\mathbf{f}[\mathbf{x}]$$

The model is a mathematical equation with a fixed form, representing a family of relations between input and output. The model contains parameters $\varphi$, where the choice of parameters determines the particular relation between input and output:

$$\mathbf{y}=\mathbf{f}[\mathbf{x},\varphi ]$$

We learn these parameters using a training dataset of $I$ pairs of input and output examples $\{x_i,y_i\}$. We aim to select parameters that map each training input to its associated output as closely as possible. We quantify the degree of mismatch in this mapping with the loss $L$. This is a scalar value that summarizes how poorly the model predicts the training outputs from their corresponding inputs for parameters $\varphi$.

We can treat the loss as a function $L[\varphi ]$ of these parameters. When we train the model, we are seeking parameters $\hat{\varphi }$ that minimize this loss function:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmin}}[L[\varphi ]]$$

If the loss is small after this minimization, we have found model parameters that accurately predict the training outputs ${\mathbf{y}}_i$ from the training inputs ${\mathbf{x}}_i$.

After training a model, we assess its performance by running the model on a separate test data to see how well it generalizes to examples that it didn’t observe during training.

Linear Regression Example

We can make the idea above more concrete with a simple example of regression. We consider a model $y=f[x,\varphi ]$ that predicts a single output $y$ from a single input $x$. A 1D linear regression model describes a straight line:

$$\begin{array}{rl}y& =f[x,\varphi ]\\ & ={\varphi }_0+{\varphi }_{1}x\end{array}$$

This model has two parameters $\varphi =[{\varphi }_0,{\varphi }_1]$, where ${\varphi }_0$ is the $y$-intercept of the line and ${\varphi }_1$ is the slope. Different choices for the intercept and the slope result in different relations, hence the model defines a family of possible input-output relations.

For this model, the training dataset consists of $I$ input/output pairs $\{x_i,y_i\}$. The mismatch between the model predictions $f[x_i,\varphi ]$ and the ground truth $y_i$. This loss is quantified using a sum of squares, over all $I$ training pairs:

$$\begin{array}{rl}L[\varphi ]& =\sum _{i=1}^I(f[x_i,\varphi ]-y_i{)}^2\\ & =\sum _{i=1}^I({\varphi }_0+{\varphi }_1{x}_i-y_i{)}^2\end{array}$$

This is called least-squares loss. The squaring operations means that the direction of the deviation (above/below the data) is unimportant.

The goal of the training process is then to find the parameters $\hat{\varphi }$ that minimize this quantity:

$$\begin{array}{rl}\hat{\varphi }& =\underset{\varphi }{\mathrm{argmin}}[L[\varphi ]]\\ & =\underset{\varphi }{\mathrm{argmin}}\left[\sum _{i=1}^I(f[x_i,\varphi ]-y_i{)}^2\right]\\ & =\underset{\varphi }{\mathrm{argmin}}\left[\sum _{i=1}^I({\varphi }_0+{\varphi }_1{x}_i-y_i{)}^2\right]\end{array}$$

There are only two parameters, so we can calculate the loss for every combination of values and visualize the loss function as a surface. The “best” parameters are at the minimum of this surface.

For training and testing, the basic method is to choose the initial parameters randomly and then improve them by “walking down” the loss function until we reach the bottom. One way to do this is to measure the gradient of the surface at the current position and take a step in the direction that is most steeply downhill. Then we repeat this process until the gradient is flat and we can improve no further.

Having trained the model, we test it by computing the loss on a separate set of test data. This shows how well the training data generalizes.

• A simple model like a line might not be able to capture the true relationship between input and output. This is known as underfitting.
• Conversely, a very expressive model may describe statistical peculiarities of the training data that are atypical and lead to unusual predictions. This is known as overfitting.