SYDE 572 A1

Assignment 1

Question 3

3-1

Prove expectation is linear: $E(aX+bY)=aE(X)+bE(Y)$.

Start with the definition of expected value. For a discrete random variable:

$$E(Z)=\sum _{z}z\cdot P(Z=z)$$

where $z$ runs over all possible values of $Z$, and $P(Z=z)$ is the probability that $Z$ takes on the value $z$. Using this definition, we have:

$$E(aX+bY)=\sum _{x,y}(ax+by)\cdot P(X=x,Y=Y)$$

Distributing the sum over the addition inside the parentheses:

$$E(aX+bY)=\sum _{x,y}ax\cdot P(X=x,Y=y)+\sum _{x,y}by\cdot P(X=x,Y=y)$$

Factor out $a$ and $b$:

$$E(aX+bY)=a\sum _{x}x\sum _{y}P(X=x,Y=y)+b\sum _{y}y\sum _{x}P(X=x,Y=y)$$

The inner sums ${\sum }_{y}P(X=x,Y=y)$ and ${\sum }_{x}P(X=x,Y=y)$ are just the marginal probabilities $P(X=x)$ and $P(Y=y)$ respectively, so we can re-write the above us:

$$\begin{array}{rl}E(aX+bY)& =a\sum _{x}P(X=x)+b\sum _{y}P(Y=y)\\ & =\boxed{aE(X)+bE(Y)}\end{array}$$

3-2

Prove variance is: ${\sigma }_x^2=E(X^2)-E(X{)}^2$.

Variance is defined as:

$$\begin{array}{rl}{\sigma }_x^2& =E[(X-{\mu }_x{)}^2]\\ & =E[X^2-2{\mu }_{x}X+{\mu }_x^2]\end{array}$$

Note that the mean, ${\mu }_x$, is a constant equal to the expected value $E(X)$. Thus, we can separate the above:

$${\sigma }_x^2=E[X^2]-2{\mu }_{x}E[X]+E[{\mu }_x^2]$$

The expectation of a constant is just the constant itself, so $E[{\mu }_x^2]={\mu }_x^2$.

$$\begin{array}{rl}{\sigma }_x^2& =E[X^2]-2{\mu }_x^2+{\mu }_x^2\\ & =E[X^2]-{\mu }_x^2\end{array}$$

Once again, ${\mu }_x=E(X)$, so:

$$\begin{array}{rl}{\sigma }_x^2& =E[X^2]-{\mu }_x^2\\ & =\boxed{E[X^2]-(E[X]{)}^2}\end{array}$$

3-3

Derive the mean and variance of $aX+b$.

Mean:

$$\begin{array}{rl}E(aX+b)& =aE(x)+b\\ & =\boxed{a{\mu }_x+b}\end{array}$$

Variance:

$$\begin{array}{rl}Var(aX+b)& =E[(aX+b-E(aX+b){)}^2]\\ & =E[(aX+b-(a{\mu }_x+b){)}^2]\\ & =E[(aX-a{\mu }_x{)}^2]\\ & =E[a^2(X-{\mu }_x{)}^2\\ & =a^2[E(X-{\mu }_x{)}^2]\end{array}$$

$E(X-{\mu }_x{)}^2$ is just the variance of $X$, ${\sigma }_x^2$. Therefore:

$$Var(aX+b)=\boxed{a^2{\sigma }_x^2}$$

3-4

Show $A$ is symmetric, where $A=V\Lambda V^{-1}$, such that $V^{-1}=V^T$.

Given:

$$\begin{array}{rl}A& =V\Lambda V^{-1}\\ V^{-1}& =V^T\end{array}$$

To show that $A$ is symmetric, we need to show that $A=A^T$. Based on the above, we have:

$$A^T=(V\Lambda V^{-1}{)}^T$$

The transpose of a product of matrices is the product of their transposes in reverse order:

$$\begin{array}{rl}{A}^T& =(V^{-1}{)}^T{\Lambda }^T{V}^T\\ & =(V^T{)}^T{\Lambda }^T{V}^T\end{array}$$

Since $\Lambda$ is a diagonal matrix, so ${\Lambda }^T=\Lambda$. Also, the transpose of the transpose of a matrix is the matrix itself, so $(V^T{)}^T=V$. So the above the expression simplifies to:

$$A^T=V\Lambda V^T$$

This is the same as $A=V\Lambda V^{-1}$, since $V^{-1}=V^T$. Thus:

$$A=A^T$$

proving that $A$ is symmetric.