# Econometric Theory/Ordinary Least Squares (OLS)

Ordinary Least Squares or OLS is one of the simplest (if you can call it so) methods of linear regression. The goal of OLS is to closely "fit" a function with the data. It does so by minimizing the sum of squared errors from the data.

## Why we Square Errors before SummingEdit

We are not trying to minimize the sum of absolute errors, but rather the sum of squared errors. Let's take a brief look at our sweater story again.

Model A

Model B

model data point error from line
A 1 5
A 2 10
A 3 -5
A 4 -10
B 1 3
B 2 -3
B 3 3
B 4 -3

Notice that the Sum of Model A is ${\displaystyle 5+10-5-10=0}$  and that the Sum of Model B is ${\displaystyle 3-3+3-3=0}$

Both Models sum to 0 and both are great fits! NO!!

So to account for the signs, whenever we sum errors, we square the terms first.

## The ModelEdit

These two models each have an intercept term ${\displaystyle \alpha }$ , and a slope term ${\displaystyle \beta }$  (some textbooks use ${\displaystyle \beta _{0}}$  instead of ${\displaystyle \alpha }$  and ${\displaystyle \beta _{1}}$  instead of ${\displaystyle \beta }$ , this is a much better approach once we move to multivariate formulas). We can represent an arbitrary single variable model with the formula: ${\displaystyle y_{i}=\alpha +\beta x_{i}+u_{i}}$  The y-values are related to the x-values given this formula. We use the subscript i to denote an observation. So ${\displaystyle y_{1}}$  is paired with ${\displaystyle x_{1}}$ , ${\displaystyle y_{2}}$  with ${\displaystyle x_{2}}$ , etc. The ${\displaystyle u_{t}}$  term is the error term, which is the difference between the effect of ${\displaystyle x_{i}}$  and the observed value of ${\displaystyle y_{i}}$ .

Unfortunately, we don't know the values of ${\displaystyle \alpha ,\beta }$  or ${\displaystyle u_{t}}$ . We have to approximate them. We can do this by using the ordinary least squares method. The term "least squares" means that we are trying to minimize the sum of squares, or more specifically we are trying to minimize the squared error terms. Since there are two variables that we need to minimize with respect to (${\displaystyle \alpha }$  and ${\displaystyle \beta }$ ), we have two equations:
${\displaystyle f=\Sigma u_{i}^{2}=\Sigma (y_{i}-\alpha -\beta x_{i})^{2}}$
${\displaystyle {\frac {\partial f}{\partial \alpha }}=-2\Sigma (y_{i}-\alpha -\beta x_{i})=0}$
${\displaystyle {\frac {\partial f}{\partial \beta }}=-2\Sigma (y_{i}-\alpha -\beta x_{i})x_{i}=0}$
Call the solutions to these equations ${\displaystyle {\hat {\alpha }}}$  and ${\displaystyle {\hat {\beta }}}$ . Solving we get:
${\displaystyle {\hat {\alpha }}={\bar {y}}-{\hat {\beta }}{\bar {x}}}$
${\displaystyle {\hat {\beta }}={\frac {\Sigma (x_{i}-{\bar {x}})y_{i}}{\Sigma (x_{i}-{\bar {x}})^{2}}}}$
Where ${\displaystyle {\bar {y}}={\frac {\Sigma y_{i}}{n}}}$  and ${\displaystyle {\bar {x}}={\frac {\Sigma x_{i}}{n}}}$ . Computing these results can be left as an exercise.

It is important to know that ${\displaystyle {\hat {\alpha }}}$  and ${\displaystyle {\hat {\beta }}}$  are not the same as ${\displaystyle \alpha }$  and ${\displaystyle \beta }$  because they are based on a single sample rather than the entire population. If you took a different sample, you would get different values for ${\displaystyle {\hat {\alpha }}}$  and ${\displaystyle {\hat {\beta }}}$ . Let's call ${\displaystyle {\hat {\alpha }}}$  and ${\displaystyle {\hat {\beta }}}$  the OLS estimators of ${\displaystyle \alpha }$  and ${\displaystyle \beta }$ . One of the main goals of econometrics is to analyze the quality of these estimators and see under what conditions these are good estimators and under which conditions they are not.

Once we have ${\displaystyle {\hat {\alpha }}}$  and ${\displaystyle {\hat {\beta }}}$ , we can construct two more variables. The first is the fitted values, or estimates of y:
${\displaystyle {\hat {y}}_{i}={\hat {\alpha }}+{\hat {\beta }}x_{i}}$
The second is the estimates of the error terms, which we will call the residuals:
${\displaystyle {\hat {u}}_{i}=y_{i}-{\hat {y}}_{i}}$
These two variables will be important later on.