Model Optimization

Given an objective function or loss function, how do we optimize it?

We have a training set of $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ of input/output pairs. We seek parameters $\varphi$ for the model $\mathbf{f}[{\mathbf{x}}_i,\varphi ]$ that maps the inputs ${\mathbf{x}}_i$ to the outputs ${\mathbf{y}}_i$ as closely as possible. To this end, we have a loss/objective function $L[\varphi ]$ that returns a single number that quantifies the mismatch in this mapping. The goal of an optimization algorithm is to find parameters $\varphi$ that minimize the loss:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmin}}[L[\varphi ]]$$

• The objective/loss function defines some surface over $\varphi$, and we want to find the $\varphi$ value at the lowest point on the surface.

Most standard optimization algorithms are iterative; they initially model parameters heuristically and then adjust them repeatedly in such a way that the loss decreases.

• Gradient Descent
  • 1D Gradient Descent
  • Multiple Dimension Gradient Descent
  • Stochastic Gradient Descent
  • Batch Gradient Descent
• Vanishing + Exploding Gradient Problem