Simple Neuron Activation Functions

Sigmoid Neuron

The activity of a neuron is very low or zero when the input is low, and the activity goes up and approaches some maximum as the input increases. This general behavior can be represented by a few activation functions.

Logistic Curve

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

Goes from 0 to 1.

• Sigmoid

Arctan

$$\sigma (z)=\arctan (z)$$

Goes from $-\frac{\pi }{2}$ to $\frac{\pi }{2}$ instead.

Hyperbolic Tangent

$$\sigma (z)=\tanh (z)$$

Goes from $-1$ to $1$.

Threshold

$$\sigma (z)=\{\begin{array}{ll}0& \text{if }z<0\\ 1& \text{if }z\ge 0\end{array}$$

This is just a Heaviside function.

Rectified Linear Unit (ReLU)

This is just a line that gets clipped below at zero:

$$\text{ReLU}(z)=\text{max}(0,z)$$

Leaky ReLU is conceptually the same but goes a bit below zero, which can have some advantages

Softmax

Softmax depends on multiple neurons. Softmax is like a probability distribution (or probability vector), so its elements add to $1$. If $\vec{z}$ is the drive (input) to a set of neurons, then:

$$\text{softmax}(\hat{z}{)}_i=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}$$

Then, by definition,

$$\sum _i\text{softmax}(\hat{z}{)}_i=1$$

so they create a probability distribution. Thus, we can turn a list of inputs (that are not a distribution) and turn them into a distribution with softmax.

One-Hot

One-Hot is the extreme of the softmax, where only the largest element ramins nonzero, while the others are set to zero. Kind of like taking the limit of the softmax?