Loss Function Recipe

To construct a loss function for training data $\{x_i,y_i\}$ using the maximum likelihood approach, we:

1. Choose a suitable probability distribution $Pr(y|\theta )$ defined over the domain of the set of predictions $\mathbf{y}$ with parameters $\theta$.
2. Set the model $f[x,\varphi ]$ to predict one or more of these parameters, so that $\theta =f[x,\varphi ]$ and $Pr(y|\theta )=Pr(y|f[x,\varphi ])$.
3. To train the model, find the network parameters $\hat{\varphi }$ that minimize the negative log-likelihood loss function over the training dataset pairs $\{x_i,y_i\}$:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmin}}[L[\varphi ]]=\underset{\varphi }{\mathrm{argmin}}\left[-\sum _{i=1}^I\log [Pr(y_i|f[x_i,\varphi ])]\right]$$

Examples of this:

• Univariate Regression as MLE
• Binary Classification
• Multi-class Classification