Convexity

In a convex function, every chord (line segment between two points on the surface) lies above the function and does not intersect it.

A surface is guaranteed to be convex if the Hessian matrix $\mathbf{H}[\varphi ]$ is positive definite (has positive eigenvalues) for all possible parameter values.

For example, for a linear regression loss surface characterized by $\varphi =[{\varphi }_0,{\varphi }_1]$, we have:

$$\mathbf{H}[\varphi ]=\left[\begin{array}{cc}\frac{{\partial }^{2}L}{\partial {\varphi }_0^2}& \frac{{\partial }^{2}L}{\partial {\varphi }_0\partial {\varphi }_1}\\ \frac{{\partial }^{2}L}{\partial {\varphi }_1\partial {\varphi }_0}& \frac{{\partial }^{2}L}{\partial {\varphi }_1^2}\end{array}\right]$$

(see UDL Chapter 6 Problems).

For any loss function, the eigenvalues of the Hessian matrix at places where the gradient is zero allow us to classify this position as:

• a minimum (the eigenvalues are all positive)
• a maximum (the eigenvalues are all negative)
• (iii) a saddle point (positive eigenvalues are associated with directions in which we are at a minimum and negative ones with directions where we are at a maximum).