Sum and Product Rules of Probability

Rules

Sum rule:

$$p(X)=\sum _{Y}p(X,Y)$$

Product rule:

$$p(X,Y)=p(Y|X)p(X)$$

Derivation

Let us have random variables $X$ and $Y$.

• $X$ can take any of the values $x_i$ where $i=1,\dots ,L$
• $Y$ can take any of the values $y_j$ where $j=1,\dots ,M$

We also define:

• $N$ is the number of trials
• $n_{ij}$ is the number of trials in which $X=x_i$ and $Y=y_j$ for $N$ trials in which we sample both of the variables $X$ and $Y$.
• $c_i$ is the number of trials in which $X$ takes the value $x_i$, irrespective of the value that $Y$ takes.
• $r_j$ is the number of trials in which $Y$ takes the value $y_j$, irrespective of the value that $X$ takes.

The joint probability of $X=x_i$ and $Y=y_j$ is written as

$$p(X=x_i,Y=y_j)=\frac{n_{ij}}{N}$$

The probability that $X$ takes the value $x_i$ irrespective of the value of $Y$ is written as $p(X=x_i)$ and is given by the fraction of the total number of points that fall in column $i$, so that:

$$p(X=x_i)=\frac{c_i}{N}$$

Because the number of instances in column $i$, denoted by $c_i$, is just the sum of the cells in the column, we have $c_i={\sum }_j{n}_{ij}$. This gives us the sum rule of probability:

Sum Rule of Probability

$$p(X=x_i)=\sum _{i=1}^{M}p(X=x_i,Y=y_j)$$

where $M$ is the number of values $Y$ can take on, such that $Y=y_j$ for $j=1,\dots ,m$.

Note that $p(X=x_i)$ is sometimes called the marginal probability and is obtained by marginalizing, or summing out, the other variables (in this case $Y$).

What if we want to consider cases where $Y=y_j$ given that $X=x_i$? This is conditional probability, which we can obtain by finding the fraction of points in column $i$ that fall in cell $i,j$:

$$p(Y=y_j|X=x_i)=\frac{n_{ij}}{c_i}$$

Using our equations from before, we can then write out the joint probability:

$$\begin{array}{rl}p(X=x_i,Y=y_j)& =\frac{n_{ij}}{N}\\ & =\frac{n_{ij}}{N}\cdot \frac{c_i}{N}\end{array}$$

This gives us the product rule of probability:

Product Rule of Probability

$$p(X=x_i,Y=y_j)=p(Y=y_j|X=x_i)p(X=x_i)$$