Davidson and MacKinnon Chapter 3—The Statistical Properties of Ordinary Least Squares
Introduction
\[\begin{eqnarray} \label{eqo1} y=X\beta+u&,& u\sim \mbox{IID}(0,\sigma^{2}I) \end{eqnarray}\]where \(y\) and \(u\) are \(n\)-vectors, \(X\) is an \(n\times k\) matrix, and \(\beta\) is a \(k\)-vector.
And the OLS estimator is
\[\begin{eqnarray} \label{eqo2} \hat{\beta}&=&(X^{T}X)^{-1}X^{T}y \end{eqnarray}\]Are OLS Estimator Unbiased?
\[\begin{eqnarray} \nonumber \hat{\beta}&=&(X^{T}X)^{-1}X^{T}(X\beta+u)\\ \label{eqo3} &=&\beta+(X^{T}X)^{-1}X^{T}u \end{eqnarray}\]Assumption:
\[E(X|u)=0\]Take conditional expectation with respect to (\ref{eqo3}), then it becomes
\[\begin{eqnarray} \nonumber E(\hat{\beta}|X)&=&\beta+E[(X^{T}X)^{-1}X^{T}u|X]\\\nonumber &=&\beta+(X^{T}X)^{-1}X^{T}E(u|X)\\ \label{eqo4} &=&\beta \end{eqnarray}\]By Law of Iterated Expectations, we have
\[\begin{eqnarray} \label{eqo5} E(\hat{\beta})&=&E\left(E(\hat{\beta}|X)\right)=E(\beta)=\beta \end{eqnarray}\]The Assumption 1 maybe too strong for time-series data. We can make following assumption
\[\begin{eqnarray} \label{eqo6} E(u_{t}|X_{t})&=&0 \end{eqnarray}\]We refer (\ref{eqo6}) as a predeterminedness condition.
Are OLS Estimator Consistent
If the sample size is large enough, the estimate will be close to the true value.
Probability Limits
\[\begin{eqnarray} \label{eqo7} \mathop{plim}_{n\to\infty} a(y^{n})&=&a_{0}\\ \label{eqo8} \lim_{n\to\infty}\Pr\left(||a(y^{n})-a_{0}||<\varepsilon\right)&=&1 \end{eqnarray}\]Law of large numbers (LLN): suppose that \(\bar{x}\) is the sample mean of \(x_{t}\), \(t=1,\dots, n\), a sequence of random variables, each with expectation \(\mu\). Then provided the \(x_{t}\) are independent, a law of large numbers would state that
\[\begin{eqnarray} \label{eqo9} \mathop{plim}_{n\to\infty}\bar{x}&=&\mathop{plim}_{n\to\infty}\frac{1}{n}\sum_{t=1}^{n}x_{t}=\mu \end{eqnarray}\]\(\bar{x}\) has a nonstochastic \mathop{plim} which is equal to the common expectation of each of the \(x_{t}\).
OLS is consistent
Assumption: \(\mathop{plim}_{n\to\infty}\frac{1}{n}X^{T}X=S_{X^{T}X}\)
| Assumption: $$E(u_{t} | x_{t})=0$$ |
The Covariance Matrix of the OLS Estimator
The full covariance matrix \(Var(b)\) can be expressed by
\[\begin{eqnarray} \label{eqo11} Var(b)&=&E\left(\left(b-E(b)\right)\left(b-E(b)\right)^{T}\right) \end{eqnarray}\]\(Var(b)\) is symmetric and positive semidefinite.
The OLS Covariance Matrix
If the error terms are IID, and have the same variance \(\sigma^{2}\), and the covariance of any pair of them is zero.
\[\begin{eqnarray} \label{eqo12} Var(u)&=&E(uu^{T})=\sigma^{2}I \end{eqnarray}\]If we assume that \(X\) is exogenous,
\[\begin{eqnarray*} E((\hat{\beta}-\beta)(\hat{\beta}-\beta)^{T}|X)&=&E((X^{T}X)^{-1}X^{T}u)(X^{T}X)^{-1}X^{T}u)^{T}|X)\\ &=&E((X^{T}X)^{-1}X^{T}uu^{T}X(X^{T}X)^{-1}|X)\\ &=&(X^{T}X)^{-1}X^{T}X(X^{T}X)^{-1}\sigma^{2}I\\ &=&\sigma^{2}(X^{T}X)^{-1} \end{eqnarray*}\]Linear Functions of Parameter Estimates
\[\begin{eqnarray*} Var(\omega^{T}\hat{\beta})&=&\omega^{T}Var(\hat{\beta})\omega\\ &=&\omega^{T}\left(\sigma^{2}(X^{T}X)^{-1}\right)\omega\\ \end{eqnarray*}\]Estimating the Variance of the Error Terms
\[\begin{eqnarray} \label{eqo13} s^{2}&=&\frac{1}{n-k}\sum_{t=1}^{n}\hat{u}_{t}^{2} \end{eqnarray}\]We obtain an unbiased estimate of \(Var(\hat{\beta})\)
\[\begin{eqnarray} \label{eqo14} Var(\hat{\beta})&=&s^{2}(X^{T}X)^{-1}=\left(\frac{1}{n-k}\sum_{t=1}^{n}\hat{u}_{t}^{2}\right)(X^{T}X)^{-1} \end{eqnarray}\]Efficiency of the OLS Estimator
Gauss-Markov Theorem: If it is assumed that \(E(u|X)=0\) and \(E(uu^{T})=\sigma^{2}I\) in the linear regression model, then the OLS estimator \(\hat{\beta}\) is more efficient than any other linear unbiased estimator \(\tilde{\beta}\), in sense that \(Var(\tilde{\beta})-Var(\hat{\beta})\) is a positive semidefinite matrix.
Goodness of Fit
Adjusted \(R^{2}\)
\[\begin{eqnarray} \label{eqo15} \bar{R}_{2}&=&1-\frac{\frac{1}{n-k}\sum_{t=1}^{n}\hat{u}_{t}^{2}}{\frac{1}{n-1}\sum_{t=1}^{n}(y_{t}-\bar{y})^{2}}=1-\frac{\frac{1}{n-k}y^{T}M_{X}y}{\frac{1}{n-1}y^{T}M_{\iota}y} \end{eqnarray}\]