Statistical Estimation

We have a probabilistic experiment with two random variables $X$ and $Y$ with marginal densities $p_X(x),p_Y(y)$. The experiment is performed, such that $x$ and $y$ are realized. However, we are only able to observe $y$. Our goal is then to estimate the unknown sample value of $x$ and $X$ from the observation $y$. In particular, we want to choose the best estimate of $x$, $\hat{X}(y)$, given $y$ and $p_{X|Y}(x|y)$.

• In this setting $p_X(s)$ is called the a priori density of $X$ and $p_{X|Y}(x|y)$ is called the a posteriori density.

Example: Estimating Noisy Binary Signal

MMSE Estimation

Given an observation $y=y_1$, find the best estimate of $x$, i.e. $\hat{x}$ such that we minimize

$$J=E[||x-\hat{x}|{|}^2|y=y_1]$$

This is called the minimum mean square error (MMSE) estimator problem.

The key idea is that to minimize the expected square error, the best choice of $\hat{x}$ is the conditional mean of $x$ given the observation:

$$\hat{x}=\mathbb{E}[x|y=y_1]$$

• The conditional mean is the best estimate in the sense that it makes the variance of the estimation error as small as possible.

Derivation

Let’s choose some arbitrary estimator $z$ (any vector or value). Then:

$$\begin{array}{rl}J& =\mathbb{E}[\Vert x-z{\Vert }^2\mid y=y_{1]}\\ & =\mathbb{E}[x^{T}x\mid y=y_1]-2z^T\mathbb{E}[x\mid y=y_1]+z^{T}z.\end{array}$$

Letting $m_{x|y_1}=\mathbb{E}[x\mid y=y_1]$, we have

$$\begin{array}{rl}J& =\mathbb{E}[x^{T}x\mid y=y_1]-2z^T{m}_{x|y_1}+z^{Tz}\\ & =\mathbb{E}[x^{T}x\mid y=y_1]+\Vert z-m_{x|y_1}{\Vert }^2-\Vert m_{x|y_1}{\Vert }^2.\end{array}$$

The only term that depends on $z$ is $\Vert z-m_{x|y_1}{\Vert }^2$. Thus, the term is minimized when $z=m_{x|y_1}=\mathbb{E}[x\mid y=y_1]$. So the optimal estimator is just

$$\hat{x}=\mathbb{E}[x|y=y_1]$$

Example: MMSE Estimation

Conditional Mean for Gaussian

For $x\sim \mathcal{N}({\mu }_x,X_{xx})$ and $y\sim \mathcal{N}({\mu }_y,X_{yy})$, the conditional pdf $p(x|y)$ is also Gaussian with its mean given by

$$\mathbb{E}[x|y]={\mu }_x+X_{xy}{X}_{yy}^{-1}(y-{\mu }_y)$$

MMSE Estimation for Gaussian