Multi-class Classification

The goal of multi-class classification is to assign an input data example $\mathbf{x}$ to one of $K>2$ classes, so the target label is $t\in \{1,2,\dots ,K\}$.

Examples:

• Predicting which of $K=10$ digits $t$ is present in an image of a handwritten number.
• Predicting which of $K$ possible words $t$ follows an incomplete sentence $\mathbf{x}$.

Following the loss function recipe, we first choose a distribution over the target space $t$. In this case, we have $t\in \{1,2,\dots ,K\}$, so we choose the categorical distribution, which is defined on this domain. This has $K$ parameters ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_K$, which determine the probability of each category:

$$P(t=k)={\lambda }_k.$$

• Constraints: Each ${\lambda }_k$ is in the range $[0,1]$ and they sum to $1$.

Then, we use a network $f[\mathbf{x},\varphi ]$ with $K$ outputs to compute these $K$ parameters from input $\mathbf{x}$. Unfortunately, the network outputs do not necessarily obey the aforementioned constraints; thus, we pass them through a function that ensures these constraints are respected. This is usually a softmax function.

The softmax takes an arbitrary vector of length $K$ and returns a vector of the same length but where the elements are now in the range $[0,1]$ and sum to $1$. The $k$-th output of the softmax function is

$${\text{softmax}}_k[\mathbf{z}]=\frac{\exp [z_k]}{{\sum }_{k^{\prime }=1}^K\exp [z_{k^{\prime }}]},$$

where the exponential functions ensure positivity, and the sum in the denominator ensures that the $K$ numbers sum to one.

The predicted probability that input $\mathbf{x}$ belongs to class $k$ is therefore

$$y_k=P(t=k\mid \mathbf{x})={\text{softmax}}_k[f[\mathbf{x},\varphi ]].$$

Given a training set $\{({\mathbf{x}}_i,t_i){\}}_{i=1}^I$, the negative log-likelihood loss is

$$\begin{array}{rl}L[\varphi ]& =-\sum _{i=1}^I\log \left[{\text{softmax}}_{t_i}\right[f[{\mathbf{x}}_i,\varphi ]\left]\right]\\ & =-\sum _{i=1}^I\left(f_{t_i}[{\mathbf{x}}_i,\varphi ]-\log \left[\sum _{k^{\prime }=1}^K\exp (f_{k^{\prime }}[{\mathbf{x}}_i,\varphi ])\right]\right).\end{array}$$

where $f_{t_i}[{\mathbf{x}}_i,\varphi ]$ and $f_{k^{\prime }}[{\mathbf{x}}_i,\varphi ]$ denote the $t_i$-th and $k^{\prime }$-th outputs of the network, respectively. This is called multiclass cross-entropy loss.

Simplified form

Let ${\mathbf{t}}^{(i)}$ be a one-hot encoding of the ground-truth class $t_i$, so that $t_{t_i}^{(i)}=1$ and every other element is zero. Then the loss can be written more compactly as

$$L[\varphi ]=-\sum _{i=1}^I\sum _{k=1}^K{t}_k^{(i)}\log y_k^{(i)},$$

where

$$y_k^{(i)}={\text{softmax}}_k[f[{\mathbf{x}}_i,\varphi ]].$$

Because the target vector is one-hot, only the correct class contributes to the loss. For the correct class, we take the negative log of its predicted probability. Every other class has a target value of zero, so its contribution to the sum is zero.

The transformed model output represents a categorical distribution over the possible classes $t\in \{1,2,\dots ,K\}$. For a point estimate, we predict the most probable class:

$$\hat{t}=\underset{k}{\mathrm{argmax}}P(t=k\mid \mathbf{x},\hat{\varphi })=\underset{k}{\mathrm{argmax}}{y}_k.$$

dl
Multiclass cross-entropy
?
Negative log of softmax simplifies to:

$$L[\varphi ]=-\sum _{i=1}^I\sum _{k=1}^K{t}_k^{(i)}\log y_k^{(i)},$$

where $t$ is a one-hot vector such that only the correct class contributes to the loss.

• $K$ classes, $I$ data samples

+++