Numerical Differentiation

Recall that any smooth function can be approximated by a Taylor series polynomial:

$$f(x)=f(a)+f^{\prime }(a)(x-a)+\frac{f^{\prime \prime }(a)}{2!}(x-a{)}^2+\cdots +\frac{f^{(n)}(a)}{n!}(x-a{)}^n+R_n$$

Forward Difference

We can use the same concept as Newton’s Divided-Difference with the Taylor series. For example, the forward Taylor series expansion can be written as:

$$f(x_{i+1})=f(x_i)+f^{\prime }(x_i)h+\frac{f^{\prime \prime }(x_i)}{2}{h}^2+\dots$$

which can be solved for:

$$f^{\prime }(x_i)=\frac{f(x_{i+1})-f(x_i)}{h}+\underset{\text{Higher order terms}}{\underbrace{\frac{h}{2}{f}^{\prime \prime }(x_i)+\frac{h^2}{3!}{f}^3(x_i)+\dots }}$$

Since $h$ is very small ($h\ll 1$), we can neglect the higher order terms. The error is then:

$$E\propto h=O(h)$$

making this a first-order approximation of order $h$.

Backward Difference

We can instead use a Taylor series for $i-1$ term:

$$f(x_{i-1})=f(x_i)-f^{\prime }(x_i)h+\frac{f^{\prime \prime }(x_i)}{2!}-\dots$$

Solving gives

$$f^{\prime }(x_i)=\frac{f(x_i)-f(x_{i-1})}{h}=\frac{\nabla f_i}{h}$$

where the error is $O(h)$ and $\nabla f_i$ is referred to as the first backward difference.

Centered Difference

Here, we use two Taylor Series for the $i+1$ and $i-1$ terms:

$$\begin{array}{rl}f(x_{i+1})& =f(x_i)+f^{\prime }(x_i)h+\frac{f^{\prime \prime }(x_i)}{2}{h}^2+\dots \\ f(x_{i-1})& =f(x_i)-f^{\prime }(x_i)h+\frac{f^{\prime \prime }(x_i)}{2!}-\dots \end{array}$$

If we subtract, we have:

$$f(x_{i+1})=f(x_{i-1})+2f^{\prime }(x_i)h+\frac{2f^{(3)}(x_i)}{3!}{h}^3+\dots$$

Solving for the first derivative:

$$f^{\prime }(x_i)=\frac{f(x_{i+1})-f(x_{i-1})}{2h}-\frac{f^{(3)}(x_i)}{6!}{h}^2-\dots$$

or:

$$f^{\prime }(x_i)=\frac{f(x_{i+1})-f(x_i)}{2h}-O(h^2)$$

This is a second order method! Less error!

Notes

Often useful for experimental data

• Fit a polynomial to the data (piecewise), and take the derivative.
• Fit a spline and take the derivative of the spline