# Numerical differentiation

**Numerical differentiation** is the set of methods for estimating the derivative of a function from sampled function values, using finite differences instead of the limiting process that defines the derivative. Because a computer cannot take a limit, every method must choose a finite step size, and the central problem of the field is balancing the two errors that step size controls: truncation error, which grows with the step, and floating-point round-off error, which grows as the step shrinks.<sup>[1](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)</sup>

| Key fact | Detail |
|---|---|
| Basic formula | Forward difference \( f'(x_0) \approx \frac{f(x_0+h)-f(x_0)}{h} \), with error \( -\frac{h}{2}f''(\xi) \), first order in \( h \)<sup>[1](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)</sup> |
| Central difference | \( \frac{f(x+h)-f(x-h)}{2h} \) has truncation error proportional to \( h^2 \)<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> |
| Optimal step | For the central difference the total error is minimized near \( h \approx \varepsilon_{\mathrm{mach}}^{1/3} \approx 10^{-5} \) in double precision, with a minimum error near \( \varepsilon_{\mathrm{mach}}^{2/3} \approx 10^{-10} \)<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> |
| Accuracy boost | Richardson extrapolation combines central differences at \( h \) and \( h/2 \) with weights \( -1/3 \) and \( 4/3 \) to raise the order from \( h^2 \) to \( h^4 \)<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> |
| Complex step | \( f'(x) \approx \mathrm{Im}\,\frac{f(x+\mathrm{i}h)}{h} \) has \( O(h^2) \) error and tolerates arbitrarily small \( h \); \( h = 10^{-100} \) has been used in practice<sup>[3](https://www.siam.org/publications/siam-news/articles/differentiation-without-a-difference/)</sup> |
| Noisy data | Differentiating measured data is typically an ill-posed problem, requiring smoothing or regularization before differences are taken<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC7899139/)</sup> |

## Finite-difference formulas

The starting point is the definition of the derivative as a limit. Choosing a small \( h \) and evaluating the difference quotient gives the forward-difference formula, whose error term, derived from the linear interpolating polynomial, is \( -\frac{h}{2}f''(\xi) \) for some \( \xi \) near \( x_0 \). Taking \( h > 0 \) gives the forward difference and \( h < 0 \) the backward difference; either way the error is first order, \( O(h) \).<sup>[1](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)</sup> As a concrete example, approximating the derivative of \( e^x \) at 0 with \( h = 0.1 \) gives 1.052, with an actual error of 0.052 against a bound of 0.0553.<sup>[1](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)</sup>

The symmetric or central difference uses points on both sides of \( x \):

\[ D_1(h) = \frac{f(x+h)-f(x-h)}{2h} = f'(x) - \frac{h^2}{6}f'''(z) \]

for some \( z \) between \( x-h \) and \( x+h \). The even-order terms in the two Taylor expansions cancel, leaving a truncation error proportional to \( h^2 \) rather than \( h \).<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> Higher-order formulas and formulas for derivatives of any order can be generated systematically; Bengt Fornberg, a mathematician known for work on spectral and finite-difference methods, published simple recursions in 1988 that compute the weights for compact finite-difference formulas of any derivative order and any accuracy order on one-dimensional grids with arbitrary spacing.<sup>[5](https://cir.nii.ac.jp/crid/1361418521390278016)</sup>

## The step-size trade-off

Differentiation is numerically unstable in a way integration is not: the difference quotient divides by a small number, so round-off eventually dominates, while integration averages errors out.<sup>[1](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)</sup> The mechanism is subtractive cancellation. If round-off perturbs the computed values of \( f(x) \) and \( f(x+h) \) by amounts of order \( \varepsilon_{\mathrm{mach}} \cdot |f(x)| \), the computed quotient carries an error of roughly \( \varepsilon_{\mathrm{mach}} |f(x)|/h \), which grows as \( h \) shrinks.<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup>

The total error is the sum of the truncation and round-off contributions, and the optimal step balances them.<sup>[6](https://www.studocu.com/en-ca/document/university-of-calgary/numerical-methods-in-engineering/chap-5-understanding-errors-in-numerical-differentiation/160595708)</sup> For the forward difference, setting \( |f''(x)|h/2 = \varepsilon_{\mathrm{mach}}|f(x)|/h \) gives an optimal \( h \) proportional to \( \varepsilon_{\mathrm{mach}}^{1/2} \), about \( 10^{-8} \) in double precision.<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> For the central difference the corresponding balance \( |f'''(x)|h^2/6 = \varepsilon_{\mathrm{mach}}|f(x)|/h \) gives \( h \) proportional to \( \varepsilon_{\mathrm{mach}}^{1/3} \), about \( 10^{-5} \), and a minimum achievable error of order \( \varepsilon_{\mathrm{mach}}^{2/3} \), about \( 10^{-10} \). In a worked example differentiating \( \sin(x) \) at \( x = 1 \), the error of the forward difference falls by a factor of 10 per decade of \( h \) down to \( h \approx 10^{-7} \) and then rises roughly in proportion to \( 1/h \); the central difference falls by a factor of 100 per decade before bottoming out at the predicted level.<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup> Methods exist that estimate an optimal step directly by minimizing the combined rounding and truncation errors.<sup>[7](https://www.sciencedirect.com/science/article/pii/0898122181900626)</sup>

## Richardson extrapolation

**Richardson extrapolation** improves a difference formula by combining evaluations at several step sizes. Because the central difference has an expansion in odd powers of \( h \), the combination

\[ D_2(h) = -\tfrac{1}{3}D_1(h) + \tfrac{4}{3}D_1(h/2) = D_1(h/2) + \frac{D_1(h/2)-D_1(h)}{3} \]

cancels the \( h^2 \) term and leaves truncation error proportional to \( h^4 \). Applying the same idea again with weights \( -1/15 \) and \( 16/15 \) cancels the \( h^4 \) term, giving \( h^6 \) accuracy. In the \( \sin(x) \) example the extrapolated error decreases by a factor of 10,000 per step before round-off ends the improvement, which illustrates why the recursion is not worth repeating too many times.<sup>[2](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)</sup>

The method is named for [Lewis Fry Richardson](https://www.edgepedia.org/lewis-fry-richardson), who introduced it in his 1911 paper on the approximate arithmetical solution of physical problems by finite differences and called it a "deferred approach to the limit"; the general procedure treats the solution of a difference problem as a function of the grid parameter \( h \), interpolates in \( h \), and evaluates the interpolant at \( h = 0 \).<sup>[8](https://encyclopediaofmath.org/wiki/Richardson_extrapolation)</sup> The special case with geometrically decreasing steps, applied to quadrature, is Romberg's rule, published by W. Romberg in 1955.<sup>[8](https://encyclopediaofmath.org/wiki/Richardson_extrapolation)</sup> A historical analysis by Hermann Engels, a mathematician at RWTH Aachen, traces the development of [Richardson extrapolation](https://www.edgepedia.org/richardson-extrapolation) and Romberg quadrature in *Historia Mathematica* (1979) and describes both as typical examples of numerically efficient and trustworthy procedures.<sup>[9](https://www.sciencedirect.com/science/article/pii/0315086079901265)</sup> Beyond polynomial interpolation, rational extrapolation in powers of \( h^p \) is also used.<sup>[8](https://encyclopediaofmath.org/wiki/Richardson_extrapolation)</sup>

## The complex-step method

For functions that are analytic, a different route avoids subtractive cancellation altogether. The complex-step approximation

\[ f'(x) \approx \mathrm{Im}\,\frac{f(x+\mathrm{i}h)}{h} \]

has error \( O(h^2) \), and because no difference of nearly equal real numbers is taken, \( h \) can be made arbitrarily small without round-off vitiating the result; the National Physical Laboratory has used \( h = 10^{-100} \). The method's first known occurrence is a 1998 *SIAM Review* paper by William Squire and George Trapp, who were inspired by earlier work of James Lyness and [Cleve Moler](https://www.edgepedia.org/cleve-moler) on complex variables for derivatives.<sup>[3](https://www.siam.org/publications/siam-news/articles/differentiation-without-a-difference/)</sup> The original paper describes a method that avoids the subtractive cancellation errors inherent in classical derivative approximations.<sup>[10](https://epubs.siam.org/doi/abs/10.1137/S003614459631241X?journalCode=sireadinfo)</sup> The trick works, in Nicholas Higham's explanation, because it acts as a form of approximate automatic differentiation with \( h \) as a symbolic variable; it requires that the computation of \( f \) itself not introduce rounding errors into the imaginary parts, and it has been adopted for sensitivity analysis in engineering and for gradient approximation in deep learning contexts.<sup>[3](https://www.siam.org/publications/siam-news/articles/differentiation-without-a-difference/)</sup> Extensions to first and second derivatives with general complex step sizes retain the absence of round-off error while improving truncation-error characteristics.<sup>[11](https://www.sciencedirect.com/science/article/pii/S0377042707004086)</sup>

## Noisy and measured data

When the input is measured data rather than an evaluable function, differentiation becomes harder in a different way. Measurements carry errors that are often much less accurate than the machine limit, and the subtraction at the heart of differencing amplifies them; the problem is typically ill-posed.<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC7899139/)</sup><sup> • </sup><sup>[12](https://arxiv.org/abs/physics/0510176)</sup>  One common approach applies a smoothing filter to the data and then takes finite differences. The classical instance is the Savitzky–Golay method, published by Abraham Savitzky and [Marcel J. E. Golay](https://www.edgepedia.org/marcel-j-e-golay) in *Analytical Chemistry* in 1964, which performs smoothing and differentiation by least-squares convolution of the data points with tabulated integer coefficients, with tables for derivatives up to fifth order using 5 to 25 points.<sup>[13](https://pubs.acs.org/doi/abs/10.1021/ac60214a047)</sup> A second family builds a local model of the data through linear regression, and a 2020 multi-objective optimization framework balances faithfulness to the data against smoothness of the derivative estimate when choosing method parameters.<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC7899139/)</sup>

## Software and alternatives

Libraries package these ideas directly. SciPy's `scipy.differentiate.derivative` performs finite-difference differentiation of black-box scalar functions, by default with an eighth-order formula, an initial step of 0.5, a step reduction factor of 2.0 per iteration, and at most 10 iterations, reusing previously evaluated function values at nested stencil points; it was inspired by the jacobi, numdifftools, and DERIVEST projects.<sup>[14](https://docs.scipy.org/doc/scipy/reference/generated/scipy.differentiate.derivative.html)</sup>

The main alternative is automatic differentiation, in which software applies the chain rule to code and computes derivatives to machine precision without a step size. JAX's `jax.grad`, for example, returns a function computing the exact gradient of a callable and can be chained for higher-order derivatives; numerical differences are then useful as an independent check on autodiff results.<sup>[15](https://docs.jax.dev/en/latest/automatic-differentiation.html)</sup> In differentiable-simulation settings, automatic differentiation avoids step-size sensitivity and delivers sensitivities at lower computational cost than finite differences, which become computationally infeasible in high-dimensional parameter spaces.<sup>[16](https://arxiv.org/abs/2606.17390)</sup> Finite differences remain the tool of choice when only a black-box function is available, and recent work examines them as an alternative to automatic differentiation in physics-informed neural networks.<sup>[17](https://arxiv.org/abs/2608.11020)</sup>

## References

1. [Per-Olof Persson, Chapter 4: Numerical Differentiation and Integration, Math 128A lecture notes, UC Berkeley](https://persson.berkeley.edu/math128a/lecture_notes/math128a_lecture_notes_chapter_4.html)
2. [James Demmel, Lecture Notes on Numerical Differentiation, Math 128a, UC Berkeley](https://people.eecs.berkeley.edu/~demmel/ma128a_Spr02/LectureNotes/Differentiation.html)
3. [Nicholas J. Higham, "Differentiation With(out) a Difference", SIAM News 51(5), 2018](https://www.siam.org/publications/siam-news/articles/differentiation-without-a-difference/)
4. [Numerical differentiation of noisy data: A unifying multi-objective optimization framework, PMC (2020)](https://pmc.ncbi.nlm.nih.gov/articles/PMC7899139/)
5. [Bengt Fornberg (1988), "Generation of finite difference formulas on arbitrarily spaced grids", Mathematics of Computation 51(184): 699–706](https://cir.nii.ac.jp/crid/1361418521390278016)
6. [Chapter 5: Understanding Errors in Numerical Differentiation, lecture notes, University of Calgary](https://www.studocu.com/en-ca/document/university-of-calgary/numerical-methods-in-engineering/chap-5-understanding-errors-in-numerical-differentiation/160595708)
7. ["Numerical estimation of the first order derivative: approximate evaluation of an optimal step", Computers & Mathematics with Applications](https://www.sciencedirect.com/science/article/pii/0898122181900626)
8. ["Richardson extrapolation", Encyclopedia of Mathematics](https://encyclopediaofmath.org/wiki/Richardson_extrapolation)
9. [Hermann Engels (1979), "Zur Geschichte der Richardson-Extrapolation", Historia Mathematica 6(3): 280–293](https://www.sciencedirect.com/science/article/pii/0315086079901265)
10. [William Squire, George Trapp (1998), "Using Complex Variables to Estimate Derivatives of Real Functions", SIAM Review 40(1): 110–112](https://epubs.siam.org/doi/abs/10.1137/S003614459631241X?journalCode=sireadinfo)
11. ["Extensions of the first and second complex-step derivative approximations", Journal of Computational and Applied Mathematics](https://www.sciencedirect.com/science/article/pii/S0377042707004086)
12. ["Numerical differentiation: local versus global methods", arXiv:physics/0510176](https://arxiv.org/abs/physics/0510176)
13. [Abraham Savitzky, Marcel J. E. Golay (1964), "Smoothing and Differentiation of Data by Simplified Least Squares Procedures", Analytical Chemistry 36(8): 1627–1639](https://pubs.acs.org/doi/abs/10.1021/ac60214a047)
14. [scipy.differentiate.derivative, SciPy v1.18.0 Manual](https://docs.scipy.org/doc/scipy/reference/generated/scipy.differentiate.derivative.html)
15. [Automatic differentiation, JAX documentation](https://docs.jax.dev/en/latest/automatic-differentiation.html)
16. ["A Differentiable GPU-Accelerated Finite Element Framework for Inverse Characterization of Finite-Strain Anisotropic Plasticity", arXiv:2606.17390](https://arxiv.org/abs/2606.17390)
17. ["Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond", arXiv:2608.11020](https://arxiv.org/abs/2608.11020)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Algorithms and computational methods › Numerical, string, and geometric algorithms › Numerical methods and approximation*

*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026; Oct 15, 2026 · Edited: Sep 30, 2026; Oct 15, 2026 · Last review: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license