Binary Classification

In binary classification, the goal is to assign the data $\mathbf{x}$ to one of two discrete classes $y\in \{0,1\}$. In this context, we refer to $y$ as a label.

Examples of binary classification:

• Predicting whether a restaurant review is positive ($y=1$) or negative ($y=0$) from text data $\mathbf{x}$
• Predicting whether a tumor is present ($y=1$) or absent ($y=0$)

We can follow the loss function recipe to construct the loss function. First, we choose a probability distribution over the output space $y\in \{0,1\}$. A suitable choice is the Bernoulli distribution, which is defined on the domain $\{0,1\}$. This has a single parameter $\lambda \in [0,1]$ that represents the probability that $y=1$:

$$Pr(y|\lambda )=\{\begin{array}{llc}1-\lambda & & y=0\\ \lambda & & y=1\end{array}$$

This can equivalently be written as

$$Pr(y|\lambda )=(1-\lambda {)}^{1-y}\cdot {\lambda }^y$$

Then, we set the model $f[\mathbf{x},\varphi ]$ to predict the single distribution parameter $\lambda$. However, $\lambda$ can only take values in the range $[0,1]$, and we cannot guarantee that the network output will lie in this range. Thus, we pass the network output through a function that maps the real numbers $\mathbb{R}$ to $[0,1]$. A suitable choice is the sigmoid:

$$\text{sig}[z]=\frac{1}{1+\exp [-z]}$$

Hence, we predict the distribution parameter as $\lambda =\text{sig}[f[\mathbf{x},\varphi ]]$. The likelihood then becomes:

$$Pr(y|\mathbf{x})=(1-\text{sig}[f[\mathbf{x},\varphi ]]{)}^{1-y}\cdot \text{sig}[f[\mathbf{x},\varphi ]{]}^y$$

This is shown below for a shallow neural network model:

The loss function is the negative log-likelihood of the training set:

$$L[\varphi ]=\sum _{i=1}^I-(1-y_i)\log [1-\text{sig}[f[{\mathbf{x}}_i,\varphi ]]]-y_i\log [\text{sig}[f[{\mathbf{x}}_i,\varphi ]]]$$

This is known as binary cross-entropy loss.

The transformed model output $\text{sig}[f[\mathbf{x},\varphi ]]$ predicts the parameter $\lambda$ of the Bernoulli distribution. This represents the probability that $y=1$, and it follows that $1-\lambda$ represents the probability that $y=0$. When we perform inference, we may want a point estimate of $y$, so we set $y=1$ if $\lambda >0.5$ and $y=0$ otherwise.