{ "id": "eprvhfcg3d", "slug": "adjoint-method", "title": "Adjoint method", "updated": "2026-09-30", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.analysis", "label": "Analysis and mathematical models", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.analysis" }, { "id": "physical.math.analysis.numerical_analysis_and_computation", "label": "Numerical analysis and computation", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.analysis.numerical_analysis_and_computation" } ], "geo": [ { "id": "geo.nongeo.t1800.physical.math.analysis", "label": "Non-geographic · 1800 to 1945: Analysis and mathematical models", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.analysis", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1800", "label": "Non-geographic · 1800 to 1945", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800" }, { "id": "geo.nongeo.t1800.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical" }, { "id": "geo.nongeo.t1800.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math" }, { "id": "geo.nongeo.t1800.physical.math.analysis", "label": "Analysis and mathematical models", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.analysis" } ] }, { "id": "geo.nongeo.t1946.physical.math.analysis.numerical_analysis_and_computation", "label": "Non-geographic · 1946 to 2000: Numerical analysis and computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.analysis.numerical_analysis_and_computation", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical" }, { "id": "geo.nongeo.t1946.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math" }, { "id": "geo.nongeo.t1946.physical.math.analysis", "label": "Analysis and mathematical models", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.analysis" }, { "id": "geo.nongeo.t1946.physical.math.analysis.numerical_analysis_and_computation", "label": "Numerical analysis and computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.analysis.numerical_analysis_and_computation" } ] } ], "excerpt": "The adjoint method computes the gradient of a scalar objective with respect to many parameters via one adjoint solve, costing 2 to 5 times one objective evaluation; in neural networks it is known as backpropagation.", "snippet": "The adjoint method computes the gradient of a scalar objective with respect to many parameters via one adjoint solve, costing 2 to 5 times one objective evaluation; in neural networks it is known as backpropagation.", "node": "physical.math.analysis.numerical_analysis_and_computation", "markdown": "# Adjoint method\n\nThe adjoint method computes the gradient of a single scalar objective function with respect to many input parameters by solving one auxiliary linear equation, the adjoint equation. Its cost is essentially independent of the parameter count: one extra solve of a system comparable to the forward problem, where finite differences would need one forward solve per parameter. Reported costs for an objective plus its full gradient are about 2 to 5 times one objective evaluation, and the gradient is exact rather than approximated.[1](https://twister.caps.ou.edu/OBAN2019/Giering_recipe4adjoint.pdf) In neural networks the same computation is known as backpropagation; in automatic differentiation it is called reverse-mode differentiation.[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)\n\n| Key fact | Detail |\n|---|---|\n| What it computes | Gradient of one scalar objective with respect to all parameters, via one adjoint solve[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf) |\n| Cost vs parameters | Almost independent of the number of model parameters; only one extra linear system is solved[3](https://adsabs.harvard.edu/pdf/2006GeoJI.167..495P) |\n| Cost vs forward run | About 2 to 5 times one objective evaluation[1](https://twister.caps.ou.edu/OBAN2019/Giering_recipe4adjoint.pdf); discrete adjoints via AD range from 2 to 20 primal simulations[4](https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume04/issue09/14371/DagRep.4.9.1/DagRep.4.9.1.pdf) |\n| Vs finite differences | At least \\( n + 1 \\) forward runs; for \\( n = 10^{6} \\) and a 1-minute simulation, 1 to 100 minutes versus almost two years[5](https://www.sciencedirect.com/science/article/pii/S0010465515004099) |\n| Linearity | The adjoint equation is always linear, even when the forward equation is not[6](https://www.oden.utexas.edu/media/reports/2023/2304.pdf) |\n| Identities | Backpropagation = method of adjoints = reverse-mode automatic differentiation[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf) • [7](https://mml-book.github.io/neurips2020/10-adjoint.pdf) |\n| Main limitation | Time-dependent problems store the forward trajectory; unchecked, a 10-minute run needs roughly a terabyte[8](https://epubs.siam.org/doi/10.1137/1.9780898717761.ch12) |\n\n## How it works\n\nFor a forward model \\( g(x, p) = 0 \\) with state \\( x \\), parameters \\( p \\), and objective \\( f(x) \\), form the Lagrangian \\( L(x, p, \\lambda) \\equiv f(x) + \\lambda^{T} g(x, p) \\), where \\( \\lambda \\) collects [Lagrange multipliers](https://www.edgepedia.org/lagrange-multipliers). Choosing \\( \\lambda \\) to satisfy the adjoint equation \\( g_{x}^{T} \\lambda = -f_{x} \\) eliminates the state sensitivity \\( x_{p} \\), leaving the gradient \\( \\frac{df}{dp} = \\lambda^{T} g_{p} \\).[9](https://cs.stanford.edu/~ambrad/adjoint_tutorial.pdf) For a linear system \\( Ax = b \\) the equivalent form is \\( \\frac{dg}{dp} = g_{p} - \\lambda^{T}(A_{p}x - b_{p}) \\).[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)\n\nThe adjoint equation is linear even for nonlinear forward models[6](https://www.oden.utexas.edu/media/reports/2023/2304.pdf), and the adjoint problem has the same size, condition number, and eigenvalue spectrum as the original system; an LU factorization \\( A = LU \\) immediately gives \\( A^{T} = U^{T}L^{T} \\), so the adjoint solve is no harder than the forward solve.[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)\n\n## How it is done\n\nThe basic optimization loop has four steps[10](https://www.crewes.org/Documents/ResearchReports/2010/CRR201091.pdf): solve the forward problem for the residuals and the objective \\( E(p) \\); solve the adjoint problem and combine it with the forward solution to obtain \\( \\nabla_{p} E(p) \\); perform a line search for a step size \\( \\alpha \\); update the model \\( p^{(k+1)} = p^{(k)} - \\alpha \\nabla_{p} E(p) \\). For time-dependent problems, one gradient requires one forward integration of the model equations over the observation window, followed by one backward integration of the adjoint equations.[11](https://ui.adsabs.harvard.edu/abs/1987QJRMS.113.1311T/abstract)\n\nDiscrete adjoints are built by automatic differentiation. During the forward evaluation a tape records the evaluation order of operators, and the gradient is computed as a matrix-free vector-Jacobian product by propagating cotangents backward through that recorded order.[12](https://pmc.ncbi.nlm.nih.gov/articles/PMC13080004/) Software implementations include dolfin-adjoint, which computes the gradient of a functional with respect to a declared Control such as an initial condition[13](https://dolfin-adjoint-doc.readthedocs.io/en/latest/documentation/tutorial.html), and differentiable simulators built on PyTorch Autograd, JAX, or source-code transformation.[14](https://mpan31415.github.io/assets/pdf/papers/2024/IEEEAccess24_DiffSim.pdf)\n\n## Origin\n\nThe adjoint method is derived from the Lagrangian dual problem, with its main ideas connected to [Pontryagin's maximum principle](https://www.edgepedia.org/pontryagins-maximum-principle)[15](http://www.nature.com/articles/s42005-024-01606-9.pdf); a related early landmark is the book *The Mathematical Theory of Optimal Processes* by L. S. Pontryagin and colleagues (1965, OR).[16](https://doi.org/10.2307/3006724) In meteorology, Talagrand and Courtier (1987, Quarterly Journal of the [Royal Meteorological Society](https://www.edgepedia.org/royal-meteorological-society)) showed that adjoint equations compute the gradient of a distance function with respect to a model's initial conditions, applying the theory to the vorticity equation with successful experiments on a Haurwitz wave.[11](https://ui.adsabs.harvard.edu/abs/1987QJRMS.113.1311T/abstract) • [17](https://doi.org/10.1002/qj.49711347812)\n\nIn geophysics the adjoint state corresponds to the backpropagated wavefield, and the gradient of the least-squares misfit acts as a migration operator.[3](https://adsabs.harvard.edu/pdf/2006GeoJI.167..495P) On the computation side, Andreas Griewank's 1992 paper in Optimization Methods & Software achieved logarithmic growth of temporal and spatial complexity in reverse automatic differentiation, the basis of checkpointing.[18](https://doi.org/10.1080/10556789208805505) In machine learning, Chen, Rubanova, Bettencourt, and Duvenaud (2018, arXiv) trained neural ordinary differential equations with the adjoint sensitivity method, solving an augmented ODE backward in time at constant memory cost.[19](https://papers.nips.cc/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf) • [20](https://doi.org/10.48550/arxiv.1806.07366)\n\n## Variants\n\n**Continuous versus discrete adjoints.** Continuous adjoint methods derive an adjoint of the primal mathematical model analytically and then discretize it; they promise a cost of about 2 primal simulations but can be mathematically challenging and numerically inconsistent with the primal discretization.[4](https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume04/issue09/14371/DagRep.4.9.1/DagRep.4.9.1.pdf) Discrete adjoints differentiate the discretized model, typically by algorithmic differentiation of the code; their cost ranges between 2 and 20 primal simulations, sometimes more, but is independent of the number of mesh points.[4](https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume04/issue09/14371/DagRep.4.9.1/DagRep.4.9.1.pdf)\n\n**Checkpointing schedules.** The offline optimal strategy, implemented in the Revolve library, is used by AD tools including ADOL-C, ADtool, Tapenade, and dolfin-adjoint.[21](https://arxiv.org/pdf/2106.13879) Multistage multilevel checkpointing was studied by Guillaume Aupy, Julien Herrmann, Paul Hovland, and Yves Robert (2016, SIAM Journal on Scientific Computing).[22](https://doi.org/10.1137/15m1019222)\n\n## Applications\n\n**Weather and ocean.** Adjoint models in meteorology and oceanography serve data assimilation, model tuning, sensitivity analysis, and singular-vector computation[1](https://twister.caps.ou.edu/OBAN2019/Giering_recipe4adjoint.pdf), and adjoint methods are described as indispensable to four-dimensional variational data assimilation (4D-Var).[12](https://pmc.ncbi.nlm.nih.gov/articles/PMC13080004/)\n\n**Seismic imaging.** [Full-waveform inversion](https://www.edgepedia.org/full-waveform-inversion) rests on the adjoint-state method, and reverse-mode automatic differentiation is mathematically equivalent to it; this equivalence underlies the ADSeismic library for velocity model estimation, rupture imaging, and source retrieval.[23](https://arxiv.org/pdf/2003.06027)\n\n**Aerodynamic design.** Adjoint design has been applied to potential flow, the Euler equations, and the [Navier–Stokes equations](https://www.edgepedia.org/navier-stokes-equations), spanning 2D airfoil, 3D wing, and complete aircraft configurations.[24](https://people.maths.ox.ac.uk/gilesm/files/ftc00.pdf) The continuous adjoint obtains complete gradient information for about twice the effort of one flow calculation, regardless of the number of design parameters.[25](https://www.sciengine.com/doi/pdf/6c4b54f6f8544412a8e2ba70f38ff0e8)\n\n**Machine learning and simulation.** Neural ODEs compute gradients through any ODE solver by the adjoint sensitivity method.[19](https://papers.nips.cc/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf) • [20](https://doi.org/10.48550/arxiv.1806.07366) Differentiable simulators, which differentiate a scalar objective with respect to many simulation parameters, use reverse-mode AD.[14](https://mpan31415.github.io/assets/pdf/papers/2024/IEEEAccess24_DiffSim.pdf)\n\n## Limitations and alternatives\n\n**Choosing a differentiation mode.** Forward (tangent-linear) algorithms are more efficient for many outputs with few parameters; adjoint algorithms for few outputs with many parameters.[26](https://ar5iv.labs.arxiv.org/html/1202.5229) Forward sensitivity analysis is quadratic-time in the number of variables, whereas [adjoint sensitivity analysis](https://www.edgepedia.org/adjoint-sensitivity-analysis) is linear.[19](https://papers.nips.cc/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf) Against finite differences, the adjoint gradient costs 2 to 5 evaluations and is exact; for \\( n = 10^{6} \\) parameters and a 1-minute simulation, adjoint AD takes 1 to 100 minutes versus at least \\( 10^{6} + 1 \\) minutes, almost two years.[1](https://twister.caps.ou.edu/OBAN2019/Giering_recipe4adjoint.pdf) • [5](https://www.sciencedirect.com/science/article/pii/S0010465515004099)\n\n**Memory.** Saving all intermediate results of a 10-minute evaluation running at several hundred million operations per second would require roughly a terabyte[8](https://epubs.siam.org/doi/10.1137/1.9780898717761.ch12), since storage grows linearly with problem size and the number of time steps.[21](https://arxiv.org/pdf/2106.13879) [Checkpointing](https://www.edgepedia.org/checkpointing) adjoints increase runtime by at most 2-fold while reducing memory to \\( N \\cdot C \\), where \\( C \\) is the number of checkpoints.[27](https://arxiv.org/pdf/1812.01892)\n\n**Failure modes.** In chaotic systems such as the Lorenz attractor, the linearized equations used in forward and adjoint computation produce solutions that blow up exponentially, so the sensitivity of a finite time average diverges.[26](https://ar5iv.labs.arxiv.org/html/1202.5229) Solving the augmented system backward can introduce instabilities absent from the forward pass, as in diffusive systems; a workaround saves additional backup states during the forward pass at higher memory cost.[15](http://www.nature.com/articles/s42005-024-01606-9.pdf) Applying AD blindly to iterative solvers such as Newton or GMRES backpropagates through every intermediate iteration, which is costly, and AD tools cannot differentiate code calling external libraries without a manually supplied vector-Jacobian product.[2](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)\n\n## References\n\n1. [Recipes for Adjoint Code Construction (Giering et al.)](https://twister.caps.ou.edu/OBAN2019/Giering_recipe4adjoint.pdf)\n2. [Notes on Adjoint Methods for 18.335 (Steven G. Johnson, MIT)](https://math.mit.edu/~stevenj/18.336/adjoint.pdf)\n3. [A review of the adjoint-state method for computing the gradient of a functional with geophysical applications](https://adsabs.harvard.edu/pdf/2006GeoJI.167..495P)\n4. [Adjoint Methods in Computational Science, Engineering, and Finance (Dagstuhl seminar report)](https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume04/issue09/14371/DagRep.4.9.1/DagRep.4.9.1.pdf)\n5. [GPU-accelerated adjoint algorithmic differentiation](https://www.sciencedirect.com/science/article/pii/S0010465515004099)\n6. [Adjoint Operators in Computational Sciences, Engineering, and Mathematics (Oden Institute Report 23-04)](https://www.oden.utexas.edu/media/reports/2023/2304.pdf)\n7. [Method of Adjoints (Machine Learning: A Probabilistic Perspective, chapter draft / NeurIPS 2020 tutorial, Ong & Deisenroth)](https://mml-book.github.io/neurips2020/10-adjoint.pdf)\n8. [Evaluating Derivatives, 2nd ed., Ch. 12: Reversal Schedules and Checkpointing (Griewank & Walther, 2008)](https://epubs.siam.org/doi/10.1137/1.9780898717761.ch12)\n9. [The adjoint method (tutorial)](https://cs.stanford.edu/~ambrad/adjoint_tutorial.pdf)\n10. [Tutorial on the continuous and discrete adjoint state method and basic implementation](https://www.crewes.org/Documents/ResearchReports/2010/CRR201091.pdf)\n11. [Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. I: Theory (Talagrand & Courtier, 1987)](https://ui.adsabs.harvard.edu/abs/1987QJRMS.113.1311T/abstract)\n12. [Fast automated adjoints for spectral PDE solvers (Dedalus)](https://pmc.ncbi.nlm.nih.gov/articles/PMC13080004/)\n13. [dolfin-adjoint tutorial: First steps](https://dolfin-adjoint-doc.readthedocs.io/en/latest/documentation/tutorial.html)\n14. [A Review of Differentiable Simulators (IEEE Access, 2024)](https://mpan31415.github.io/assets/pdf/papers/2024/IEEEAccess24_DiffSim.pdf)\n15. [Adjoint methods for parameter estimation in differential equations (adoptODE tutorial, Communications Physics 2024)](http://www.nature.com/articles/s42005-024-01606-9.pdf)\n16. [M. L. Chambers and colleagues (1965). The Mathematical Theory of Optimal Processes. OR.](https://doi.org/10.2307/3006724)\n17. [Olivier Talagrand, Philippe Courtier (1987). Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. I: Theory. Quarterly Journal of the Royal Meteorological Society.](https://doi.org/10.1002/qj.49711347812)\n18. [Andreas Griewank (1992). Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization methods & software.](https://doi.org/10.1080/10556789208805505)\n19. [Neural Ordinary Differential Equations (NeurIPS 2018)](https://papers.nips.cc/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf)\n20. [Chen, Ricky T. Q. and colleagues (2018). Neural Ordinary Differential Equations. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1806.07366)\n21. [Optimal checkpointing for adjoints of multistage time-stepping schemes (CAMS)](https://arxiv.org/pdf/2106.13879)\n22. [Guillaume Aupy and colleagues (2016). Optimal Multistage Algorithm for Adjoint Computation. SIAM Journal on Scientific Computing.](https://doi.org/10.1137/15m1019222)\n23. [A General Approach to Seismic Inversion with Automatic Differentiation (ADSeismic)](https://arxiv.org/pdf/2003.06027)\n24. [An Introduction to the Adjoint Approach to Design (Giles & Pierce)](https://people.maths.ox.ac.uk/gilesm/files/ftc00.pdf)\n25. [Continuous adjoint method for aerodynamic design optimization (external and internal flows)](https://www.sciengine.com/doi/pdf/6c4b54f6f8544412a8e2ba70f38ff0e8)\n26. [Forward and Adjoint Sensitivity Computation of Chaotic Dynamical Systems](https://ar5iv.labs.arxiv.org/html/1202.5229)\n27. [Performance and Scalability of Discrete Local Sensitivity Analysis via Automatic Differentiation versus Continuous Adjoint Sensitivity Analysis](https://arxiv.org/pdf/1812.01892)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Analysis and mathematical models › Numerical analysis and computation*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://math.mit.edu/~stevenj/18.336/adjoint.pdf", "https://cs.stanford.edu/~ambrad/adjoint_tutorial.pdf" ], "url": "https://www.edgepedia.org/adjoint-method", "markdown_url": "https://www.edgepedia.org/adjoint-method.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Adjoint method\", Edgepedia (EdgeChat), https://www.edgepedia.org/adjoint-method. Edgepedia Community License 1.0.", "credit_md": "\"[Adjoint method](https://www.edgepedia.org/adjoint-method)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/adjoint-method](https://www.edgepedia.org/adjoint-method). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Adjoint method\", Edgepedia (EdgeChat), https://www.edgepedia.org/adjoint-method. Edgepedia Community License 1.0.", "speakable": "The adjoint method computes the gradient of a scalar objective with respect to many parameters via one adjoint solve, costing 2 to 5 times one objective evaluation; in neural networks it is known as backpropagation." }