Click for printer friendely version of this HowTo Proof of $F$-Distribution The goal here is to show that Equation 3.13.21, that is $$\frac {({\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta})' \... ...at{\boldsymbol{\beta}})'({\bf Y} - {\bf X}\hat{\boldsymbol{\beta}})/ (n - p)},$$ has an $F$-distribution. We will do this by showing that the numerator and the denominator are both $\sigma^2$ times chi-square variables divided by their degrees of freedom. Since $\hat{\boldsymbol{\beta}} = ({\bf X}'{\bf X})^{-1}{\bf X}'{\bf Y}$ is a linear function of Y, and ${\bf Y} \sim N({\bf X}\boldsymbol{\beta}, \sigma^2)$, a vector of $n$ iid random variables, it follows from Equations 3.13.7 and 3.13.8 that $\hat{\boldsymbol{\beta}}$ is a vector of $p$ random variables with a $N(\boldsymbol{\beta}, \sigma^2[{\bf X}'{\bf X}]^{-1})$ distribution. Thus, the transformation, ${\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta}$, results in $t$ random variables ($t$ being the number of rows in C, the number of tests): $${\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta} \sim N_t(... ...\beta} - \boldsymbol{\theta}, \sigma^2{\bf C}[{\bf X}'{\bf X}]^{-1}{\bf C}').$$ Under the hypothesis that ${\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta} = 0$ we have, $${\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta} \sim N_t(0, \sigma^2{\bf C}[{\bf X}'{\bf X}]^{-1}{\bf C}'),$$ thus, $$[{\bf C}({\bf X}'{\bf X})^{-1}{\bf C}']^{-1/2}({\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta}) \sim N_t(0, \sigma^2)$$ $$\sigma^2[{\bf C}({\bf X}'{\bf X})^{-1}{\bf C}']^{-1/2}({\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta}) \sim N_t(0, 1).$$ (D.7.1) Since the sum of $t$ squared iid $N(0, 1)$ variables results in a random variable distributed by $\chi^2_t$, it follows from Equation D.7.1 that, $$({\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta})' [{\bf C}... ...{\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta}) \sim \sigma^2\chi^2_t.$$ Thus, we have shown that the numerator in Equation 3.13.21 is $\sigma^2$ times a chi-square random variable divided by its degrees of freedom. Showing the same thing for the denominator is a little more tricky as it involves some obscure transformations and knowing a few properties of quadratic forms. Instead of trying to explain the details about quadratic forms that would be required for a full proof, we'll simply go as far as we can with what we have and appeal to your sense of intuition. Let $\hat{{\bf e}} = {\bf Y} - {\bf X}\hat{\boldsymbol{\beta}}$, an approximation of the error vector, $\hat{\boldsymbol{\epsilon}}$, thus, $$\hat{{\bf e}} = {\bf Y} - {\bf X}\hat{\boldsymbol{\beta}}$$ $$= {\bf Y} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}'{\bf Y}$$ $$= ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}'){\bf Y}$$ $$= ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')({\bf X}\boldsymbol{\beta} + \boldsymbol{\epsilon})$$ $$= ({\bf X}\boldsymbol{\beta} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}... ...beta}) + ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')\boldsymbol{\epsilon}$$ $$= ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')\boldsymbol{\epsilon},$$ and $$({\bf Y} - {\bf X}\hat{\boldsymbol{\beta}})'({\bf Y} - {\bf X}\hat{\boldsymbol{\beta}}) = [({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')\boldsymbol{\epsilon}]' [({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')\boldsymbol{\epsilon}]$$ $$= \boldsymbol{\epsilon}'({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')' ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')\boldsymbol{\epsilon}$$ (D.7.2) $$= \boldsymbol{\epsilon}' ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\... ...bf X})^{-1}{\bf X}'{\bf X}({\bf X}'{\bf X})^{-1}{\bf X}') \boldsymbol{\epsilon}$$ (D.7.3) $$= \boldsymbol{\epsilon}' ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\... ...X})^{-1}{\bf X}' + {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}') \boldsymbol{\epsilon}$$ (D.7.4) $$= \boldsymbol{\epsilon}' ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}') \boldsymbol{\epsilon}.$$ (D.7.5) Equations D.7.2, D.7.3, D.7.4 and D.7.5 make it clear that $({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')$ is idempotent, that is, $$({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')'({\bf I} - {\bf... ... X}'{\bf X})^{-1}{\bf X}') = ({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}').$$ We can determine the rankD.1 of $({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')$ from its trace (that is, the sum the elements on the diagonal) since it is idempotent and symmetric. Using a well known property of traces, that is, tr$(ABC) =$ tr$(CAB)$, and the fact that X is an $n \times p$ matrix, we have $$\mathrm{tr}({\bf I}_n - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}') = \mathrm{tr}({\bf I}_n) - \mathrm{tr}({\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')$$ $$= n - \mathrm{tr}({\bf X}'{\bf X}({\bf X}'{\bf X})^{-1})$$ $$= n - \mathrm{tr}({\bf I}_p)$$ $$= n - p.$$ Since $\boldsymbol{\epsilon} \sim N_n(0, \sigma^2)$, it follows that $\boldsymbol{\epsilon}/\sigma \sim N_n(0, 1)$, thus we can imagine that Equation D.7.5 is the sum of $n - p$ independent normal random variables. Thus, if we let $\lambda \sim N(0, 1)$, then $$\sum_{i=1}^{n-p} \epsilon_i\epsilon_i = \sum_{i=1}^{n-p} \sigma\lambda\sigma\lambda$$ $$= \sum_{i=1}^{n-p} \sigma^2\lambda^2 \sim \sigma^2\chi^2_{n-p}$$ Showing that the numerator is independent from the denominator also requires some obscure transformations and requires another result from quadratic forms. Without proof, I will state that the following theorem. Let Z is a vector of of normally distributed random variables with a common variance and let $q_1 = {\bf Z}'{\bf A}{\bf Z}$ and $q_2 = {\bf Z}'{\bf B}{\bf Z}$, where A and B are both $n \times n$ symmetric matrices. $q_1$ and $q_2$ are independently distributed if and only if ${\bf AB} = 0$. Now, to show independence, under the hypothesis that C $\boldsymbol{\beta} = \boldsymbol{\theta}$, can re-write the ends of the numerator with $$({\bf C}\hat{\boldsymbol{\beta}} - \boldsymbol{\theta}) = {\bf C}\hat{\boldsymbol{\beta}} - {\bf C}\boldsymbol{\beta}$$ $$= {\bf C}({\bf X}'{\bf X})^{-1}[{\bf X}'{\bf Y} - {\bf X}'{\bf X}\boldsymbol{\beta}]$$ $$= {\bf C}({\bf X}'{\bf X})^{-1}{\bf X}'[{\bf Y} - {\bf X}\boldsymbol{\beta}]$$ $$= {\bf C}({\bf X}'{\bf X})^{-1}{\bf X}'\boldsymbol{\epsilon}.$$ If we let ${\bf A} = {\bf X}({\bf X}'{\bf X})^{-1}{\bf C}\left[ {\bf C}({\bf X}'{\bf X})^{-1}{\bf C} \right]^{-1}{\bf C}({\bf X}'{\bf X})^{-1}{\bf X}'$, then we can rewrite the numerator of Equation 3.13.21 as $\boldsymbol{\epsilon}'{\bf A}\boldsymbol{\epsilon}$. If we let ${\bf B} = {\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}'$, then we can rewrite the denominator to be $\boldsymbol{\epsilon}'{\bf B}\boldsymbol{\epsilon}$. Since $${\bf X}'{\bf B} = {\bf X}({\bf I} - {\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')$$ $$= {\bf X}' - {\bf X}'{\bf X}({\bf X}'{\bf X})^{-1}{\bf X}')$$ $$= {\bf X}' - {\bf X}'$$ $$= 0,$$ ${\bf AB} = 0$, and thus, under the hypothesis, the numerator and denominator are independent. Click for printer friendely version of this HowTo

>