Code Verification in Scientific Computing: Methods and Worked Steps

Code verification, in scientific computing, is the process of checking that a simulation or analysis program correctly implements the mathematical model and numerical algorithms its developers intended. It asks “are we solving the equations right?” and it is answered by comparing program output with known solutions, checking that numerical error shrinks at the rate theory predicts, and testing the building blocks of the code. The most widely taught general-purpose procedure is the method of manufactured solutions (MMS) combined with an observed-order-of-accuracy test, walked through step by step below.

The term is overloaded. This page uses the research-software sense, defined by the AIAA guide G-077-1998 as part of verification: “the process of determining that a model implementation accurately represents the developer’s conceptual description” (as quoted in the NASA Glenn NPARC verification and validation tutorial). Other senses are separated in the first table so you can tell quickly whether this is the page you need.

What code verification means (and what it does not)

The National Academies report Assessing the Reliability of Complex Models (2012, chapter 3) describes verification as the process of determining how accurately a computer program solves the equations of a model, and splits it into code verification (does the program correctly implement the intended algorithms?) and solution verification (how large is the numerical error in a specific simulation?). Code verification is a property of the software; it says nothing about whether the model equations describe nature. That second question is validation, a separate activity.

Phrase “code verification” used for	What is being checked	Is it this page?
Scientific and engineering simulation codes	That the program implements the intended mathematical model and numerical scheme correctly (AIAA G-077 / National Academies sense)	Yes
Phone or e-mail one-time codes	That a person controls a number or address	No
Blockchain contract “source verification”	That published source code matches deployed bytecode	No
Software-engineering quality gates (CI/CD tools, static analysis)	Generic defects and style rules in application code	Partly; unit testing overlaps, but the numerical-accuracy tests below are specific to scientific computing

The last three rows describe how the phrase is used in search results generally; they are listed only to disambiguate and are not claims about any particular vendor.

What it is used for in research

Research groups use code verification wherever a result depends on a numerical solver: finite-element and finite-volume models, computational fluid dynamics, ODE and PDE integrators, geodynamics, climate and reservoir models. A bug in such a code rarely crashes the program. It silently produces a smooth, plausible and wrong answer, which is why verification is built around quantitative tests rather than inspection of plots.

NASA Glenn’s tutorial on verification and validation of CFD codes, citing Roache’s framing, describes code verification as evaluating errors such as bugs, incorrect implementations of conceptual models, and errors in inputs. For funders and reviewers who ask for evidence that a published computational result can be trusted, a documented verification suite is the artefact that answers the question.

Code verification versus solution verification

The two are easy to conflate. The National Academies chapter treats them separately:

• Code verification (section 3.2) determines whether the program correctly implements the intended algorithms. It uses comparison with analytic solutions and the method of manufactured solutions, and its building blocks include unit tests that assess whether the fundamental pieces of the code correctly execute their intended algorithms.
• Solution verification (section 3.3) estimates the accuracy of the numerical error in specific quantities of interest for one given simulation. It is not a statement about general code correctness.

In practice you do code verification once per code version and problem class, on problems where the right answer is known, and solution verification for every production run, where it is not.

Methods of code verification

The NASA Glenn NPARC tutorial lists the following methods for assessing a CFD code, which generalise to other numerical software. The grid and time-step convergence studies produce the observed order of convergence, error bands and the grid convergence index (GCI).

Method (NASA Glenn list)	What you do
Code examination	Review the source for bugs by visual inspection and by testing subprograms
Iterative convergence	Confirm that iterative solvers converge as expected
Consistency checks	Check conserved quantities, for example mass conservation
Spatial (grid) convergence	Refine the mesh and examine how error changes
Temporal convergence	Refine the time step and examine how error changes
Comparison with highly accurate solutions	Compare with analytical or benchmark ODE/PDE solutions, with a second code that uses different numerical methods, or against suites of highly accurate cases

The weakness of comparing with analytic solutions is availability: numerical methods exist because analytic solutions are hard or impossible to find. NTNU course notes on MMS make the same point and use it to motivate manufacturing a solution instead.

Install or access

Code verification is a procedure, not a product, so there is nothing to install for the method itself. The two worked sources used here need the following.

• Python route (NTNU notes). The example scripts import NumPy, Matplotlib and SymPy (from sympy import symbols, diff, lambdify, sin), plus the course’s own ODE scheme module. Obtain each package from its official project documentation.
• Finite-element route (PyLith). The MMS tests are part of PyLith’s developer test suite, documented at pylith.readthedocs.io under developer testing. The example suite lives in the PyLith source tree at tests/mmstests/linearelasticity/nofaults-2d.

Step-by-step: method of manufactured solutions and order of accuracy

The following sequence follows the NTNU course notes on MMS, with the order-of-accuracy test they use as the acceptance criterion. Suppose the code solves D y = f(x, y), where D is a differential operator.

1. Choose a manufactured solution y_m. Pick any smooth function, for example sin(x). The notes point out it should exercise the equation: for a third-order ODE, choosing y_m = x would make the second and third derivatives zero and test almost nothing, so they use a Gaussian instead. All derivatives up to the order of the equation should be continuous.
2. Compute the manufactured source term. f_m = D y_m − f(x, y_m). The notes do this with SymPy (f_m = diff(y_m, x) - beta*y_m for dy/dx = λy with y_m = sin(x), giving f_m = cos(x) − λ sin(x)).
3. Solve the modified problem. The program now solves D y = f + f_m, with boundary and initial conditions taken from y_m (for the example, y(x₀) = sin(x₀)). By construction y_m is the exact solution of this modified problem, so no external reference is needed.
4. Compute the error on successively refined grids. The notes use the infinity norm, max|y − y_m|, with the time-step count doubled each pass (N = 10, 20, 40, and so on, five refinements).
5. Compute the observed order. If the error behaves as ε = C h^p and the refinement ratio is r, then p = log(ε₁/ε₂) / log(r). With r = 2 this becomes p = log(ε_previous/ε_current) / log 2, which is the line np.log(previous_max_error/max_error)/np.log(2) in the notes’ script.
6. Compare with the theoretical order. The acceptance criterion in the notes is that the observed order of accuracy is close to the theoretical order of the scheme.

Worked reading of the formula: with r = 2, an error that halves when the step halves gives p = 1 (first order); an error that falls to a quarter gives p = 2 (second order). That is arithmetic on the formula above, not a result taken from a source.

Output and how to read it

The NTNU notes verify three ODE integrators (Euler, Heun and a fourth-order Runge-Kutta scheme) on the third-order ODE y′′′ + y′′ y + y′ = 0 with a Gaussian manufactured solution. They fit ε = C h^p to two step sizes and report the following results; the data table at the end of this page sets them beside the PyLith test criteria.

Scheme	Reported fit	Theoretical order	Reading
Euler	ε = 2.19 h1.00	1	Matches
Heun	ε = 4.5 h2.00	2	Matches
Fourth-order Runge-Kutta	ε = 0.19 h3.908	4	Very close to 4; the notes call all three observed orders very close to theoretical

A plot of observed order against number of time steps should settle on horizontal lines at the theoretical values; the notes draw dotted lines at 1, 2 and 4 for this. An observed order clearly below the theoretical one signals an implementation error, a boundary-condition error or a solution outside the asymptotic range where ε ≈ C h^p holds. The notes explicitly assume C and p are constant only in that asymptotic range.

Finite-element codes: the four PyLith MMS tests

PyLith’s developer documentation applies MMS to finite-element code in four tests, implemented in the C++ class pylith::testing::MMSTest. The page says the process is: assume a solution, substitute it into the governing equations, and treat the leftover residual as a body force added so the assumed solution becomes exact.

1. Solution representation. The solution must be representable in the finite-element space; a polynomial of order 2 needs a basis order of at least 2.
2. Residual. For the known solution the residual must be zero within a tolerance, ‖F(s) − G(s)‖ ≤ ε, where F and G are the left- and right-hand-side residuals.
3. Taylor-series Jacobian. The remainder ‖F(s + εδs) − F(s) − εJv‖ must be smaller than ε², which shows the Jacobian gives the expected order of accuracy.
4. Finite-difference Jacobian. The finite-difference Jacobian must agree with the hand-coded Jacobian within a tolerance.

You must supply analytic functions for every solution and auxiliary subfield, Dirichlet boundary conditions, a mesh and the problem setup. The documentation warns: “The quadrature order must be the same across all solution subfields and auxiliary subfields.” Its example is the TestLinearElasticity suite with the UniformStrain2D case in tests/mmstests/linearelasticity/nofaults-2d. The page does not describe a mesh-refinement convergence study, so treat these four tests as complementary to, not a replacement for, the order-of-accuracy procedure above.

Troubleshooting failed tests

Residual test fails

1. Confirm the discretization check passes.
2. Run only the residual test for one discretization with the debug flag --journal.debug=mms-test.
3. Check that the residual kernels appear in the PETSc discretization view.
4. Find the degrees of freedom with nonzero residual terms and use the solution section to identify their subfield and point.
5. Review the pointwise residual and solution functions for that subfield.

Jacobian test fails

1. Confirm the residual check passes first.
2. Run only the finite-difference Jacobian test for one discretization with --journal.debug=mms-test.
3. Compare the hand-coded Jacobian with the finite-difference Jacobian.
4. Check the PETSc discretization view for expected pointwise functions that are missing.
5. Remove terms from the residual, Jacobian and solution pointwise functions until the tests pass, then add them back one at a time, fixing errors as each appears.

Observed order is wrong in an ODE or PDE script

Apply the same isolate-and-restore logic: simplify the manufactured solution so fewer terms of the operator are active, confirm the source term by differentiating with SymPy rather than by hand (the NTNU notes note the source term for even a modest nonlinear ODE becomes a very long expression), and confirm that the manufactured solution actually exercises every term of the operator.

Free alternatives and tools named in the sources

Tool	Role in code verification	Where documented
SymPy (Python)	Symbolic differentiation to build the manufactured source term and lambdify it for NumPy	NTNU MMS course notes
PyLith MMS test framework	Residual and Jacobian tests for finite-element codes built on PETSc	PyLith developer documentation, MMS tests

Related CASRAI pages on the surrounding tooling: numpy.allclose for tolerance comparisons of arrays in tests, coverage.py for measuring how much of a Python code base a test suite exercises, CasADi as an example of a numerical optimisation framework, and software of unknown provenance for the regulated-software view of unverified components. Software metadata for citing the verified version can be recorded with CodeMeta.

FAQ

What is code verification in scientific computing?

It is the process of determining that a computer program correctly implements the intended algorithms and model equations, as distinct from validation, which tests whether the model represents reality. Definitions come from AIAA G-077-1998 and the National Academies 2012 report.

What is the difference between code verification and solution verification?

Code verification checks the program on problems with known answers. Solution verification estimates numerical error for a specific simulation and quantity of interest, and says nothing about general code correctness (National Academies, sections 3.2 and 3.3).

What is the method of manufactured solutions?

You choose a solution, substitute it into the governing equations, and add the leftover residual to the equations as a source term so that the chosen solution is exact. Then you check that the code reproduces it and that error falls at the theoretical rate as the grid is refined.

How do you calculate the observed order of accuracy?

With error ε = C h^p and refinement ratio r, p = log(ε₁/ε₂)/log(r). Compare p with the scheme’s theoretical order.

Do I need an analytic solution?

No. That is the reason MMS exists: the NTNU notes contrast it with comparison against an exact solution, which requires one to exist.

Is passing one test enough?

NASA Glenn lists several complementary methods, from code examination and consistency checks to grid and time-step convergence and comparison with benchmark solutions; verification evidence normally combines more than one.

Sources

• NASA Glenn Research Center, NPARC Alliance CFD verification and validation tutorial: Verification assessment (read 2026-11-13).
• National Academies of Sciences, Engineering, and Medicine, Assessing the Reliability of Complex Models (2012), chapter 3 (read 2026-11-13).
• PyLith developer documentation, Method of manufactured solutions tests (read 2026-11-13).
• NTNU course notes, Code verification using the Method of Manufactured Solutions (read 2026-11-13).