Softmax

Takes a whole vector $Z\in {\mathbb{R}}^n$ and generates as output a vector $A\in [0,1]$ with the property that ${\sum }_{i=1}^n{A}_i=1$, which means we can interpret it as a probability distribution over $n$ items:

$$\text{softmax}(z)=\left[\begin{array}{c}\exp (z_1)/{\sum }_i\exp (z_i)\\ ⋮\\ \exp (z_n)/{\sum }_i\exp (z_i)\end{array}\right]$$

Softmax is similar to sigmoid in concept (used for outputting probabilities/confidences) but in higher dimensions. Commonly used for multi-class classification.

Stable Softmax

Naive implementation:

def softmax(items_in):
    exps = np.exp(items_in)
    items_out = exps / np.sum(exps)
    return items_out

This has issues with numerical stability. If any element of items_in is large, then np.exp(items_in) will overflow.

Numerically stable implementation:

def softmax(items_in):
    shifted = items_in - np.max(items_in)
    exps = np.exp(shifted)
    return exps / np.sum(exps)

This works because subtracting the same constant $c$ from every input does not change the output of the softmax:

\frac{e^{x_i-c}}{\sum_j e^{x_j-c}}
=
\frac{e^{x_i}}{\sum_j e^{x_j}}

Proof:

$$\begin{array}{rl}\frac{e^{x_i-c}}{{\sum }_j{e}^{x_j-c}}& =\frac{e^{x_i}{e}^{-c}}{{\sum }_j{e}^{x_j}{e}^{-c}}\\ & =\frac{e^{x_i}{e}^{-c}}{e^{-c}{\sum }_j{e}^{x_j}}\\ & =\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}\end{array}$$

dl Numerically stable softmax ? Subtracting the same constant $c$ from every input does not change the output of the softmax. So we subtract the maximum element to prevent overflow.

def softmax(items_in):
    shifted = items_in - np.max(items_in)
    exps = np.exp(shifted)
    return exps / np.sum(exps)

+++