Skip to main content
v2026.11,986 entries · CC-BY 4.0

PETSc: Scientific Computing & Solver Guide

A complete guide to PETSc for research, covering installation from source, parallel linear solver (KSP) setup, output diagnostics, and troubleshooting.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

PETSc (Portable, Extensible Toolkit for Scientific Computation) is an open-source suite of data structures and numerical routines designed for the scalable parallel solution of scientific applications modeled by partial differential equations (PDEs), nonlinear equations, and large sparse linear systems. Developed primarily at Argonne National Laboratory under a 2-Clause BSD license, PETSc provides high-performance building blocks for computational science using the Message Passing Interface (MPI) standard and modern GPU architectures.

What it is used for in research

Researchers in computational fluid dynamics, structural mechanics, geophysics, plasma physics, and astrophysics use PETSc to solve large-scale discretized continuum models that exceed the memory and processing capabilities of single-node workstations. PETSc structures computational simulations into modular abstractions:

  • Parallel Vectors (Vec) and Matrices (Mat): Distributed data structures that manage parallel partitioning, vector norms, and sparse compressed row (AIJ) matrix storage across thousands of MPI ranks.
  • Linear Solvers (KSP) and Preconditioners (PC): Scalable Krylov subspace methods (such as Conjugate Gradients, GMRES, and BiCGStab) coupled with algebraic multigrid (PCGAMG), block Jacobi, and domain decomposition preconditioners.
  • Nonlinear Equation Solvers (SNES): Scalable Newton-based algorithms, including line search, trust region, and pseudo-transient continuation methods for nonlinear PDE systems.
  • Time Integrators (TS): Explicit, implicit, and IMEX (implicit-explicit) ordinary differential equation (ODE) and differential-algebraic equation (DAE) time-stepping schemes with adjoint sensitivity capabilities.
  • Data Management (DM): Abstractions such as DMDA for structured Cartesian meshes and DMPlex for unstructured finite volume or finite element meshes that connect geometrical discretizations directly to solver data structures.
  • Optimization Solvers (TAO): The Toolkit for Advanced Optimization provides unconstrained, bound-constrained, and PDE-constrained optimization algorithms for parameter estimation and design synthesis.

In multi-physics simulation and engineering research, PETSc serves either as a direct programming foundation or as the backend solver engine beneath higher-level finite element and multiphysics frameworks, complementing commercial simulation packages such as COMSOL Multiphysics in large cluster deployments.

How to install or access it

PETSc is distributed directly from its official GitLab repository and documentation portal maintained by Argonne National Laboratory. Source installation is the standard and recommended approach on high-performance computing clusters and research workstations to ensure optimal compiler vectorization and MPI binding.

1. Prerequisites

Ensure the following build prerequisites are installed on your host system:

  • C99 and C++11 compliant compilers (e.g., GCC, Clang, or Intel oneAPI)
  • Python version 3.6 or higher (required to execute PETSc’s configuration engine)
  • Git version control and GNU Make
  • CMake (required if downloading certain third-party external solver packages)

2. Clone the Release Source

Clone the stable release branch from the official PETSc GitLab repository:

git clone -b release https://gitlab.com/petsc/petsc.git petsc
cd petsc

3. Configuration

Run PETSc’s Python-based configuration utility. If your research environment already includes a vendor-optimized MPI implementation and BLAS/LAPACK libraries, run:

./configure

If you are working on a standalone development workstation without pre-installed parallel libraries, instruct PETSc to automatically download and build MPICH and BLAS/LAPACK:

./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack

To reduce build times when your application code does not require Fortran interoperability (and does not depend on Fortran-only external solvers such as MUMPS), disable the Fortran interface explicitly:

./configure --with-fc=0 --with-fortran-bindings=0

For production research benchmark runs, disable runtime debugging symbols and activate full compiler optimizations:

./configure --with-debugging=0 COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3"

4. Compilation and Verification

Following a successful configuration, compile the PETSc shared libraries and run the verification test suite:

make all
make check

The make check command compiles and executes a test solver, confirming that MPI process spawning, matrix-vector kernels, and convergence checks function correctly.

Step-by-step instructions

This workflow demonstrates writing, compiling, and executing a parallel linear solver task in C using PETSc’s KSP module to solve the system A x = b across multiple MPI ranks.

  1. Initialize the PETSc Environment:

    Include the primary solver header <petscksp.h> and initialize MPI and runtime argument tables inside main() using PetscInitialize():

    #include <petscksp.h>
    
    int main(int argc, char **args) {
        Vec x, b;
        Mat A;
        KSP ksp;
        PetscInt n = 10000;
        
        PetscFunctionBeginUser;
        PetscCall(PetscInitialize(&argc, &args, NULL, "Solve parallel linear system."));
  2. Create and Preallocate the Parallel Sparse Matrix:

    Instantiate a distributed sparse matrix object using MatCreate() and specify local and global problem dimensions. Preallocate non-zero storage using MatMPIAIJSetPreallocation() to eliminate runtime dynamic heap reallocations:

        PetscCall(MatCreate(PETSC_COMM_WORLD, &A));
        PetscCall(MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, n, n));
        PetscCall(MatSetFromOptions(A));
        PetscCall(MatMPIAIJSetPreallocation(A, 3, NULL, 2, NULL));
        PetscCall(MatSeqAIJSetPreallocation(A, 3, NULL));
  3. Assemble Matrix Values and Complete Parallel Assembly:

    Populate matrix entries (such as a 1D discrete Laplacian stencil) and synchronize across all MPI processes using two-phase assembly calls:

        PetscInt Istart, Iend, i;
        PetscScalar v;
        PetscCall(MatGetOwnershipRange(A, &Istart, &Iend));
        for (i = Istart; i < Iend; i++) {
            v = -1.0;
            if (i > 0) PetscCall(MatSetValue(A, i, i - 1, v, INSERT_VALUES));
            if (i < n - 1) PetscCall(MatSetValue(A, i, i + 1, v, INSERT_VALUES));
            v = 2.0;
            PetscCall(MatSetValue(A, i, i, v, INSERT_VALUES));
        }
        PetscCall(MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY));
        PetscCall(MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY));
  4. Create Problem Vectors:

    Allocate parallel solution vector x and right-hand side vector b compatible with matrix A:

        PetscCall(MatCreateVecs(A, &x, &b));
        PetscCall(VecSet(b, 1.0));
        PetscCall(VecAssemblyBegin(b));
        PetscCall(VecAssemblyEnd(b));
  5. Configure and Solve with KSP:

    Create the linear solver context, link the system matrix operator, inject command-line runtime options with KSPSetFromOptions(), and solve:

        PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp));
        PetscCall(KSPSetOperators(ksp, A, A));
        PetscCall(KSPSetFromOptions(ksp));
        PetscCall(KSPSolve(ksp, b, x));
  6. Free Resources and Finalize:

    Destroy all allocated PETSc handles prior to calling PetscFinalize() to ensure clean memory termination:

        PetscCall(KSPDestroy(&ksp));
        PetscCall(MatDestroy(&A));
        PetscCall(VecDestroy(&x));
        PetscCall(VecDestroy(&b));
        PetscCall(PetscFinalize());
        return 0;
    }
  7. Compile and Execute in Parallel:

    Compile using PETSc’s compiler wrapper or CMake rules incorporating CTest for regression verification, then execute across 4 MPI processes with runtime solver arguments:

    mpiexec -n 4 ./linear_solve -ksp_type cg -pc_type jacobi -ksp_monitor -log_view

Output and how to read it

When running with -ksp_monitor and -log_view, PETSc emits real-time iteration histories followed by a comprehensive performance profiling report to standard output.

1. KSP Convergence Monitor Output

The -ksp_monitor flag prints residual reductions at each Krylov iteration:

  0 KSP Residual norm 1.000000000000e+02
  1 KSP Residual norm 4.714045207910e+01
  2 KSP Residual norm 1.839210492100e+01
  3 KSP Residual norm 5.120349182310e+00
  4 KSP Residual norm 1.204918239012e-01
  5 KSP Residual norm 8.948210491820e-05
  6 KSP Residual norm 4.129381029410e-09

Each row details:

  • First column: The iteration count index k, starting from initial guess 0.
  • Second column: The solver stage indicator (KSP Residual norm).
  • Third column: The preconditioned Euclidean 2-norm of the residual vector, ‖B-1(b – A xk)‖2. In well-conditioned problems, this value decays monotonically toward your specified relative tolerance (default -ksp_rtol 1e-5) or absolute tolerance (default -ksp_atol 1e-50).

2. Logging and Performance Diagnostics (-log_view)

The -log_view profiling summary breaks down computational resource utilization across all participating MPI ranks:

--- Event Stage 0: Main Stage

Phase                Count     Time (sec)        Flops/sec        --- Global ---
                             Max     Ratio      Max     Ratio   %T %F %M %R %M
MatMult               12   0.024s    1.02     1.42e+09   1.01    15 32 20 18 10
KSPSolve               1   0.142s    1.01     2.15e+09   1.02    88 95 90 92 85
PCSetUp                1   0.008s    1.04     0.00e+00   0.00     5  0  0  0  0
PCApply                7   0.019s    1.01     4.10e+08   1.01    12 18 15 14 12
VecDot                 7   0.005s    1.08     9.80e+08   1.05     3  8  8 10 12

Key metrics to examine in the profile:

  • Time Ratio: Ratio of maximum time spent by any single MPI process to the minimum time. A ratio near 1.0 indicates balanced computation across ranks, whereas ratios above 1.5 signal load imbalance or communication stalls.
  • Flops/sec: Sustained floating-point throughput per process during matrix-vector products (MatMult) and preconditioner applications (PCApply).
  • MPI Reductions (%R): Percentage of total MPI all-reduce synchronizations, primarily originating from inner product computations (VecDot, VecNorm) in Krylov subspace projections.

Troubleshooting

Common operational and algorithmic issues encountered during PETSc development, along with their verified remediations from official documentation:

1. Matrix Assembly Memory Bottleneck (Slow MatSetValues)

Symptom: The program pauses for extended periods or memory usage balloons during the MatSetValue() or MatSetValues() loop before assembly completes.

Cause: PETSc matrices allocate internal CSR storage dynamically if non-zero entries per row have not been preallocated. Each unexpected non-zero triggers heap reallocation and memory copying.

Fix: Calculate or overestimate the maximum number of diagonal and off-diagonal non-zero entries per row and call MatSeqAIJSetPreallocation() (for sequential matrices) or MatMPIAIJSetPreallocation() (for parallel matrices) immediately after MatSetSizes().

2. Linear Solver Divergence (DIVERGED_ITS or DIVERGED_DTOL)

Symptom: KSPSolve() fails with reason DIVERGED_ITS (exceeded maximum iterations) or DIVERGED_DTOL (residual exceeded divergence tolerance).

Cause: Either the operator is ill-conditioned, the preconditioner is inadequate for the spectral radius, or an inappropriate solver was selected (e.g., using the Conjugate Gradient method -ksp_type cg on an unsymmetric or indefinite matrix).

Fix:
Switch to a generalized Krylov method such as GMRES (-ksp_type gmres) or BiCGStab (-ksp_type bcgs). Strengthen the preconditioning strategy by switching from point Jacobi to algebraic multigrid (-pc_type gamg) or domain decomposition (-pc_type asm). Check solver progress at the command line without modifying source code:

mpiexec -n 4 ./linear_solve -ksp_type gmres -pc_type gamg -ksp_converged_reason

3. Segmentation Fault on Process Finalization

Symptom: Application completes its numerical computations successfully but crashes with a memory violation or signal 11 during termination.

Cause: Calling VecDestroy(), MatDestroy(), or KSPDestroy() after PetscFinalize() has already released internal MPI communicators and memory pools. In C++, this frequently occurs when PETSc wrapper objects reside in main() scope and invoke their destructors after PetscFinalize().

Fix: Explicitly destroy all PETSc objects before calling PetscFinalize(), or encapsulate solver handles inside an inner lexical block { ... } so destructors execute prior to finalization.

Free alternatives

The table below summarizes leading open-source scientific computing and numerical solver frameworks available under permissive licenses, suitable for peer-reviewed research published in venues like the Journal of Open Source Software (JOSS).

Framework Developer / Origin License Distributed Parallelism Primary Research Strengths
PETSc Argonne National Laboratory 2-Clause BSD MPI, CUDA, HIP, SYCL, Kokkos Scalable parallel sparse linear (KSP) and nonlinear (SNES) solvers, ODE/DAE time integration (TS), and mesh abstraction (DM) for PDE simulations and topology optimization.
Trilinos Sandia National Laboratories Modified BSD MPI, Kokkos, OpenMP Modular C++ package ecosystem (Tpetra, Belos, Amesos2, Ifpack2, Anasazi) for multiphysics simulations, eigenproblems, and automatic differentiation.
SUNDIALS Lawrence Livermore National Laboratory BSD 3-Clause MPI, OpenMP, Pthreads, CUDA, HIP Specialized, high-precision stiff and non-stiff ODE integrators (CVODE, ARKODE), differential-algebraic equations (IDA), and sensitivity analysis.
Eigen Open-source Community MPL 2.0 Shared-memory (No native distributed MPI) Header-only C++ template library for dense and sparse matrix algebra, geometry transforms, and local workstation simulations.

FAQ

What is the difference between PETSc and standard linear algebra libraries like BLAS and LAPACK?

BLAS and LAPACK provide low-level dense matrix-vector operations and sequential direct matrix factorizations. PETSc operates at a higher abstraction level, providing distributed-memory sparse matrix structures, iterative Krylov solvers, scalable preconditioners, and mesh management on top of underlying BLAS/LAPACK and MPI routines.

Can PETSc be used with Python?

Yes. PETSc includes first-class Python bindings known as petsc4py. It allows researchers to construct parallel vectors, matrices, and KSP/SNES solvers using Python and NumPy arrays while retaining native C execution speeds across distributed MPI processes.

What license does PETSc use?

PETSc is released under the permissive 2-Clause BSD license (FreeBSD License). It permits free academic, governmental, and commercial redistribution, modification, and integration into both open-source and proprietary software stacks, as defined in standard open-source software guidelines.

How do I switch linear solvers in PETSc without recompiling my code?

Ensure your program calls KSPSetFromOptions() after KSPCreate(). You can then pass solver and preconditioner configurations directly at execution time via the terminal, such as -ksp_type gmres -pc_type ilu -ksp_max_it 500.

Sources

  • PETSc Official Documentation: Quick Start Tutorial (Release 3.26.1). https://petsc.org/release/install/install_tutorial/ (read 2026-11-11)
  • PETSc Official Documentation: Architecture and Library Overview (Release 3.26.1). https://petsc.org/release/overview/ (read 2026-11-11)
  • PETSc Official Documentation: KSP Linear System Solvers Manual (Release 3.26.1). https://petsc.org/release/manual/ksp/ (read 2026-11-11)
  • PETSc Official Documentation: Frequently Asked Questions and Troubleshooting (Release 3.26.1). https://petsc.org/release/faq/ (read 2026-11-11)

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about PETSc: Scientific Computing & Solver Guide

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →