Bayes' Theorem

Definition

Bayes’ Theorem states:

$$p(Y|X)=\frac{p(X|Y)p(Y)}{p(X)}$$

Derivation

The product rule tells us:

$$p(X,Y)=p(Y|X)p(X)$$

The symmetry property tells us:

$$\begin{array}{rl}p(X,Y)& =p(Y,X)\\ & =p(X|Y)p(Y)\end{array}$$

Thus, we have:

$$\begin{array}{rl}p(Y|X)p(X)& =p(X|Y)p(Y)\\ p(Y|X)& =\frac{p(X|Y)p(Y)}{p(X)}\end{array}$$

Notes

Bayes’ theorem relates the conditional distribution $p(Y|X)$ to the reversed conditional distribution $p(X|Y)$.

Using the sum rule, the denominator in Bayes’ theorem can be expressed by the quantities appearing in the numerator:

$$p(X)=\sum _{Y}p(X|Y)p(Y)$$

Thus, we can view the denominator as being a normalization constant, ensuring that the sum over the conditional probability distribution $p(Y|X)$ on the left hand side of Bayes’ theorem over all $Y$ values equals $1$.