Click for printer friendely version of this HowTo Properties of $\hat{\boldsymbol{\beta}}$ If we assume that the elements in the noise vector, $\boldsymbol{\epsilon}$, are independent and normally distributed 3.15random variables with $\mu_{\epsilon} = 0$ and $\sigma^2_{\epsilon} = \sigma^2$, (which is not terribly unreasonable to do since noise can come from all kinds of sources and once we add them all up, the central limit theorem kicks into effect,) then we can determine if $\hat{\boldsymbol{\beta}}$ is biased and what its variance is. Before we start, however, we will note that the assumption that $\boldsymbol{\epsilon}$ are independent and identically distributed normal(0, $\sigma^2$) variables implies that Y is also normally distributed with mean $\bf {X}\boldsymbol{\beta}$ and variance $\sigma^2$. This is because ${\bf Y} = {\bf X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$ and ${\bf X}\boldsymbol{\beta}$ functions as a location parameter. First, we will show that $\hat{\boldsymbol{\beta}}$ is unbiased. $${\bf E}\hat{\boldsymbol{\beta}} = {\bf E}\left\{\mathbf{[X'X]^{-1}X'Y}\right\}$$ $$= \mathbf{[X'X]^{-1}X'\left(EY\right)}$$ $$= \mathbf{[X'X]^{-1}X'\left(X\boldsymbol{\beta}\right)}$$ $$= \mathbf{[X'X]^{-1}X'X\boldsymbol{\beta}}$$ $$= \boldsymbol{\beta}.$$ (3.13.7) Now we will derive the variance of $\hat{\boldsymbol{\beta}}$. However, before we get into it, let me first point out that E $(\boldsymbol{\epsilon}\boldsymbol{\epsilon}') = \sigma^2$. This can be easily shown using the facts that Var $(\boldsymbol{\epsilon}) = \sigma^2$, E $\boldsymbol{\epsilon} = 0$ and the definition of variance. That is, $${\bf Var}(\boldsymbol{\epsilon}) = {\bf E}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}') - {\bf E}\boldsymbol{\epsilon}{\bf E}\boldsymbol{\epsilon}'$$ $$= {\bf E}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}') - 0$$ $$\sigma^2 = {\bf E}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}').$$ With that little bit of extra information in hand, we are now ready to derive the variance of $\hat{\boldsymbol{\beta}}$. $${\mathbf{Var(\hat{\boldsymbol{\beta}})}}$$ $$= {\bf E} \left\{ ( \hat{\boldsymbol{\beta}} - {\bf E} \hat{\boldsy... ...\beta}}) {\bf (\boldsymbol{\hat{\beta}} - E \hat{\boldsymbol{\beta}})'}\right\}$$ $$= {\bf E} \left\{ [ {\bf (X'X)^{-1}X'Y} - \boldsymbol{\beta}] [{\bf (X'X)^{-1}X'Y} - \boldsymbol{\beta}]'\right\}$$ $$= {\bf E} \left\{ [{\bf (X'X)^{-1}X}'({\bf X}\boldsymbol{\beta} + \boldsymbol{\epsilon}) - \boldsymbol{\beta}] \right.$$ $$\left. [{\bf (X'X)^{-1}X}'({\bf X}\boldsymbol{\beta} + \boldsymbol{\epsilon}) - \boldsymbol{\beta}]' \right\},$$ $$\textrm{since } {\bf Y} = {\bf X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$ $$= {\bf E} \left\{ [ \boldsymbol{\beta} + {\bf (X'X)^{-1}X}'\boldsymbol{\epsilon} - \boldsymbol{\beta}]\right.$$ $$\left. [ \boldsymbol{\beta} + {\bf (X'X)^{-1}X}'\boldsymbol{\epsilon} - \boldsymbol{\beta}]' \right\}$$ $$= {\bf E} \left\{ [{\bf (X'X)^{-1}X}'\boldsymbol{\epsilon}] [{\bf (X'X)^{-1}X}'\boldsymbol{\epsilon}]'\right\}$$ $$= \mathbf{ E\left\{(X'X)^{-1}X' \boldsymbol{\epsilon} \boldsymbol{\epsilon} ' X(X'X)^{-1}\right\} }$$ $$= \mathbf{ (X'X)^{-1} X' E(\boldsymbol{\epsilon} \boldsymbol{\epsilon} ') X(X'X)^{-1}}$$ $$= \mathbf{ (X'X)^{-1} X'X(X'X)^{-1}}\sigma^2$$ $$= \mathbf{(X'X)^{-1}}\sigma^2.$$ An alternative and shorter derivation of this same variance is as follows: $${\mathbf{Var(\hat{\boldsymbol{\beta}})}}$$ $$= {\bf Var}({\bf (X'X)^{-1}X'Y})$$ $$= [{\bf (X'X)^{-1}X'}]{\bf Var}({\bf Y})[{\bf (X'X)^{-1}X'}]'$$ $$= \mathbf{ (X'X)^{-1} X'X(X'X)^{-1}}\sigma^2$$ $$= \mathbf{(X'X)^{-1}}\sigma^2.$$ (3.13.8) Click for printer friendely version of this HowTo

>