Explicit Regularization

Consider fitting a model $\mathbf{f}[\mathbf{x},\varphi ]$ with parameters $\varphi$ using a training set $\{{\mathbf{x}}_i,{\mathbf{y}}_i\}$ of input/output pairs. We seek the parameters $\hat{\varphi }$ that minimize the loss function $L[\varphi ]$:

$$\begin{array}{rl}\hat{\varphi }& =\underset{\varphi }{\mathrm{argmin}}[L[\varphi ]]\\ & =\underset{\varphi }{\mathrm{argmin}}\left[\sum _{i=1}^I{\ell }_i[{\mathbf{x}}_i,{\mathbf{y}}_i]\right]\end{array}$$

where the individual terms ${\ell }_i[{\mathbf{x}}_i,{\mathbf{y}}_i]$ measure the mismatch between the network predictions $\mathbf{f}[{\mathbf{x}}_i,\varphi ]$ and output targets ${\mathbf{y}}_i$ for each training pair. To bias this minimization toward certain solutions:

$$\hat{\varphi }=\underset{\varphi }{\mathrm{argmin}}\left[\sum _{i=1}^I{\ell }_i[{\mathbf{x}}_i,{\mathbf{y}}_i]+\lambda \cdot g[\varphi ]\right]$$

where:

• $g[\varphi ]$ is a function that returns a scalar which takes larger values when the parameters are less preferred.
• $\lambda$ is a positive scalar that controls the relative contribution of the original loss function and the regularization term.

The minima of the regularized loss function usually differ from those in the original, so the procedure converges to different parameter values.