Polynomial Regression

We extend least squares criterion to fit polynomials with higher orders.

For example, a second order polynomial:

y = a_{0} + a_{1} x + a_{2} x^{2} + e

In this case, the sum of squares of residuals would be:

S_{r} = i = 1 \sum n (y_{i} - a_{0} - a_{1} x_{i} - a_{2} x_{i}^{2})^{2}

Taking the derivative with respect to the coefficients:

\frac{\partial S _{r}}{\partial a _{0}} \frac{\partial S _{r}}{\partial a _{1}} \frac{\partial S _{r}}{\partial a _{1}} = - 2 \sum (y_{i} - a_{0} - a_{1} x_{i} - a_{2} x_{i}^{2}) = - 2 \sum x_{i} (y_{i} - a_{0} - a_{1} x_{i} - a_{2} x_{i}^{2}) = - 2 \sum x_{i}^{2} (y_{i} - a_{0} - a_{1} x_{i} - a_{2} x_{i}^{2})

Setting these equal to $0$ and rearranging gives:

n a_{0} + (\sum x_{i}) a_{1} + (\sum x_{i}^{2}) a_{2} (\sum x_{i}) a_{0} + (\sum x_{i}^{2}) a_{1} + (\sum x_{i}^{3}) a_{2} (\sum x_{i}) a_{0} + (\sum x_{i}^{3}) a_{1} + (\sum x_{i}^{4}) a_{2} = \sum y_{i} = \sum x_{i} y_{i} = \sum x_{i}^{2} y_{i}

We can rewrite this into form:

[A] ⎩ ⎨ ⎧ a_{0} a_{1} a_{2} ⎭ ⎬ ⎫ = {b}

which we can then solve with a method like Gauss-Seidel or Gaussian Elimination.

Note that this extends to polynomials of any order $m$ as long as the $m$ is less than the number of data points $n$ .

So, the steps are:

/notes/