Click for printer friendely version of this HowTo Parameter Estimation: The Least Squares Method Given data for the dependent and independent variables, X and Y, how should we estimate the values for $\boldsymbol{\beta}$, the model parameters? For this we can use the least squares procedure. That is, estimate $\boldsymbol{\beta}$ by minimizing the total squared differences between observed and predicted values. The difference between the observed and predicted values, often times called the residual, is, in matrix notation, ${\bf Y - X}\boldsymbol{\beta}$. The squared residual is $({\bf Y - X}\boldsymbol{\beta})' ({\bf Y - X}\boldsymbol{\beta})$. Thus, $$\mathbf{F}(\boldsymbol{\beta}) = (\mathbf{Y - X \boldsymbol{\beta})' (Y - X \boldsymbol{\beta}})$$ $$= \mathbf{Y'Y - Y'X\boldsymbol{\beta} - \boldsymbol{\beta}'X'Y + \boldsymbol{\beta}'X'X\boldsymbol{\beta}}.$$ (3.13.4) To minimize Equation 3.13.4, we take its derivative with respect to $\boldsymbol{\beta}$, set it equal to zero and solve for $\boldsymbol{\beta}$. $${\frac{\partial}{\partial \beta} \mathbf{Y'Y - Y'X\boldsymbol{\beta} - \boldsymbol{\beta}'X'Y + \boldsymbol{\beta}'X'X\boldsymbol{\beta}}}$$ $$= \mathbf{-X'Y - X'Y + 2 X'X \boldsymbol{\beta}}$$ $$= \mathbf{-2X'Y + 2 X'X \boldsymbol{\beta} \stackrel{\mathrm{set}}{=} 0},$$ and thus3.14 $$\boldsymbol{\hat{\beta}} = \mathbf{[X'X]^{-1}X'Y}.$$ (3.13.5) If we substitute our estimated parameters, $\boldsymbol{\hat{\beta}}$, into Equation 3.13.4, we get the following simplification for calculating the squared residual: $${{\bf F(\boldsymbol{\hat{\beta}})}}$$ $$= {\bf Y'Y - Y'X\boldsymbol{\hat{\beta}} - \boldsymbol{\hat{\beta}}'X'Y + \boldsymbol{\hat{\beta}}'X'X\boldsymbol{\hat{\beta}}}$$ $$= {\bf Y'Y - Y'X\boldsymbol{\hat{\beta}} - \boldsymbol{\hat{\beta}}'X'Y + \boldsymbol{\hat{\beta}}'X'X(X'X)^{-1}X'Y}$$ $$= {\bf Y'Y - Y'X\boldsymbol{\hat{\beta}} - \boldsymbol{\hat{\beta}}'X'Y + \boldsymbol{\hat{\beta}}'X'Y}$$ $$= {\bf Y'Y - Y'X\boldsymbol{\hat{\beta}}}.$$ (3.13.6) Click for printer friendely version of this HowTo

>