Loss Function

A loss function $L[\varphi ]$ quantifies the quality of a prediction $f(x_i,\varphi )$ compared to the ground truth $y_i$. During training, we seek parameter values $\varphi$ that minimize the loss, mapping the training inputs to outputs as closely as possible.

Loss functions return a single scalar value.

A basic example is 0-1 loss:

$$L_{01}(h(x;\Theta ),y)=\{\begin{array}{ll}0& \text{if }y=h(x;\Theta )\\ 1& \text{otherwise}\end{array}$$