Maximum Likelihood Criterion

Under the conditional probabilistic perspective of learning, the model now computes different distribution parameters ${\theta }_i=f[x_i,\varphi ]$ for each training input $x_i$.

Each observed training output $y_i$ should have high probability under its corresponding distribution $Pr(y_i|{\theta }_i)$. Hence, we choose the model parameters $\varphi$ so that they maximized the combined probability across all $I$ training samples:

$$\begin{array}{rl}\hat{\varphi }& =\underset{\varphi }{\mathrm{argmax}}\left[\prod _{i=1}^{I}Pr(y_i|x_i)\right]\\ & =\underset{\varphi }{\mathrm{argmax}}\left[\prod _{i=1}^{I}Pr(y_i|{\theta }_i)\right]\\ & =\underset{\varphi }{\mathrm{argmax}}\left[\prod _{i=1}^{I}Pr(y_i|f[x_i,\varphi ])\right]\end{array}$$

The combined probability term is the likelihood of the parameters. Thus, the above is known as the maximum likelihood criterion.

A more practical version of the maximum likelihood criterion is the Log-Likelihood Criterion.