Least Squares Regression

Least-squares regression seeks to minimze the residual (or error) between data points and corresponding points on the curve (trend line).

Consider the equation of a straight line:

$$y=\underset{\text{intercept}}{\underbrace{a_0}}+\underset{\text{slope}}{\underbrace{a_1}}x+\underset{\text{error}}{\underbrace{e}}$$

Rearranging, we would have:

$$e=y-a_0-a_{1}x$$

A common strategy would be to minimize the sum of squares of the residuals, such as:

$$S_r=\sum _{i=1}^n{e}_i^2=\sum _{i=1}^n(y_{i,\text{measured}}-y_{i,\text{model}})=\sum _{i=1}^n(y_i-a_0-a_1{x}_i{)}^2$$

Least Squares Regression Formulation

We differentiate the sum of residuals with respect to the coefficients:

$$\begin{array}{rl}\frac{\partial S_r}{\partial a_0}& =-2\sum (y_i-a_0-a_1{x}_i)\\ \frac{\partial S_r}{\partial a_1}& =-2\sum [(y_i-a_0-a_1{x}_i)x_i]\end{array}$$

Setting the derivatives equal to zero will minimize $S_r$:

$$\begin{array}{rl}0& =\sum y_i-\sum a_0-\sum a_1{x}_i\\ 0& =\sum y_i-\sum a_0{x}_i-\sum a_1{x}_i^2\end{array}$$

Since $\sum a_0=n{a}_0$, where $n$ is the number of data points, we can express the equations as a set of simultaneous linear equations with two unknowns:

$$\begin{array}{rl}n{a}_0+\left(\sum x_i\right)a_1& =\sum y_i\\ \left(\sum x_i\right)a_0+\left(\sum x_i^2\right)a_1& =\sum x_i{y}_i\end{array}$$

\Thus, we can solve for $a_0,a_1$:

Least Square Regression Equations

$$\begin{gathered} a_1=\frac{n\sum x_i{y}_i-\sum x_i\sum y_i}{n\sum x_i^2-{\left(\sum x_i\right)}^2} \\ {a}_0=\bar{y}-a_1\bar{x} \end{gathered}$$

where $\bar{y},\bar{x}$ are the means of $y,x$