Covariance

For two random variables $x$ and $y$, the covariance measures the extent to which the two variables vary together. This is defined by:

$$\begin{array}{rl}\text{cov}[x,y]& =E_{x,y}\left[(x-E[x])(y-E[y])\right]\\ & =E_{x,y}[xy]-E[x]E[y]\end{array}$$

If $x$ and $y$ are independent, then their covariance equals zero.

For two vectors $\mathbf{x}$ and $\mathbf{y}$, their covariance is a matrix given by:

$$\begin{array}{rl}\text{cov}[\mathbf{x},\mathbf{y}]& =E_{\mathbf{x},\mathbf{y}}\left[\{\mathbf{x}-E[\mathbf{x}]\}\{{\mathbf{y}}^T-E[{\mathbf{y}}^T]\}\right]\\ & =E_{\mathbf{x},\mathbf{y}}[{\mathbf{xy}}^T]-E[\mathbf{x}]E[{\mathbf{y}}^T]\end{array}$$

If we consider the covariance of the components of a vector $\mathbf{x}$ with each other, then we use a slightly simpler notation $\text{cov}[x]\equiv \text{cov}[x,x$].