Written and maintained by CASRAI Editorial Board
Published
On this page
Performance profiles are a standardized statistical methodology and visual diagnostic used in numerical optimization and scientific computing to compare the computational performance of multiple algorithms or software solvers across a common benchmark set of test problems. First formulated by Elizabeth Dolan and Jorge Moré in 2002, performance profiles resolve the distortions of raw execution averages by plotting cumulative distribution functions of solver performance ratios, enabling researchers to objectively assess solver efficiency, convergence speed, and algorithmic robustness without allowing a single catastrophic failure or extreme outlier to skew evaluation metrics.
What it is used for in research
Empirical algorithm evaluation is fundamental across computational operations research, numerical analysis, machine learning, and engineering design. When researchers develop novel mathematical solvers or evaluate existing software libraries—such as modeling frameworks in CVXPY, root-finding packages in scipy fsolve, or discrete solvers in mixed integer programming—they must demonstrate computational advantage across heterogeneous test suites. Historically, benchmarking reports relied on summary statistics such as arithmetic mean runtimes, cumulative CPU times, or geometric mean speedups across problem collections. However, these aggregate measures suffer from severe mathematical vulnerabilities.
Arithmetic averages are excessively sensitive to performance outliers on ill-conditioned problem instances. A solver that performs well on 95% of a test suite but stalls on the remaining 5% will show an artificially inflated mean runtime. Conversely, cumulative CPU time allows a single problem that runs for hours to dominate aggregate rankings. Furthermore, when evaluating algorithms with different asymptotic behavior as studied in computational complexity, absolute runtimes scale nonlinearly with problem dimension, conflating problem scale with algorithmic capability.
A further limitation of traditional statistics is the treatment of solver failures. Numerical solvers frequently fail due to iteration limits, numerical overflow, or time limits. Assigning an arbitrary penalty time distorts averages, making solver comparisons dependent on the chosen penalty constant. Dolan-Moré performance profiles overcome these limitations by establishing relative performance ratios for each individual problem, eliminating dimensional scaling effects and treating solver failures as points that sit beyond the finite evaluation threshold.
Mathematical foundations and ratio formulation
To construct a performance profile, researchers define a set of benchmark test problems denoted as P and a set of candidate solvers denoted as S, with cardinalities |P| = n_p and |S| = n_s. For each problem p in P and solver s in S, the researcher records a strictly positive performance metric t_{p,s} > 0, representing execution time, iteration counts, or objective function evaluations. Smaller values indicate superior performance.
The baseline performance for problem p is defined by the best performance achieved by any solver in the candidate set: min { t_{p,s’} : s’ in S }. The relative performance ratio r_{p,s} evaluates solver s against this optimal baseline:
r_{p,s} = t_{p,s} / min { t_{p,s’} : s’ in S }
By definition, r_{p,s} ≥ 1 for all problems and solvers. If solver s is the fastest algorithm on problem p, then r_{p,s} = 1. If solver s requires three times as much computational effort as the best-performing solver on that specific instance, r_{p,s} = 3. When solver s fails to solve problem p, the ratio is set to r_{p,s} = r_M, where r_M is an upper penalty parameter satisfying r_M ≥ max { r_{p,s} : r_{p,s} < ∞ }, or formally r_{p,s} = ∞.
The performance profile of solver s is the empirical cumulative distribution function (CDF) of its performance ratios across the problem set P:
ρ_s(τ) = (1 / |P|) × size { p in P : r_{p,s} ≤ τ }
Here, ρ_s(τ) represents the probability that the performance ratio r_{p,s} is within a factor τ ≥ 1 of the best possible performance. Following the Dolan-Moré convention, ρ_s(τ) is plotted as a function of log2(τ). On this scale, a horizontal axis value of 0 corresponds to τ = 1 (the best solver), a value of 1 corresponds to τ = 2, and a value of 2 corresponds to τ = 4. This logarithmic transformation ensures that proportional performance differences receive equal visual weight across the plotting domain.
Install or access (official source only)
Researchers can generate publication-quality performance profiles using dedicated open-source packages available across major scientific computing environments. In Python, two primary packages provide official, maintained implementations:
- OptiProfiler: Developed by Tom M. Ragonneau and Zaikun Zhang, OptiProfiler benchmarks derivative-free and derivative-based optimization algorithms. It generates Dolan-Moré performance profiles, data profiles, and log-ratio profiles using Matplotlib. Install via PyPI:
pip install optiprofiler - perprof-py: Developed by Abel Soares Siqueira and colleagues, perprof-py is an open-source Python package for generating Dolan-Moré profiles from CSV data or programmatic matrices, supporting Matplotlib PNG, PGF/TikZ for LaTeX inclusion, and Bokeh HTML output. Install via PyPI:
pip install perprof-py
Standard scientific Python dependencies—NumPy, SciPy, Matplotlib, and Pandas—should be installed in your environment to facilitate data manipulation and matrix operations:
pip install numpy scipy matplotlib pandas
In Julia, the JuliaSmoothOptimizers organization maintains BenchmarkProfiles.jl. Install directly through Julia’s package manager:
using Pkg
Pkg.add("BenchmarkProfiles")
For MATLAB and GNU Octave users, the classic perfprof.m script developed alongside Higham and Higham’s MATLAB Guide provides a lightweight plotting utility for performance profile matrices.
Step-by-step task
The following walkthrough illustrates the complete process of executing a multi-solver benchmark, assembling performance matrices, computing Dolan-Moré performance ratios, and generating publication-ready performance profile graphics in Python.
- Step 1: Collect raw benchmark metrics into a structured matrix
Run your benchmark test suite across all candidate solvers under identical hardware and operating system conditions. Record the measured cost metric (e.g., wall-clock execution time in seconds) in a two-dimensional matrix or Pandas DataFrame where each row represents a unique problem instance p and each column represents a solver s. Mark failures or timeouts usingnp.nanornp.inf. - Step 2: Apply a numerical floor to prevent zero-division artifacts
Microsecond runtimes can produce distorted ratios when solving trivial problem instances. Apply an explicit numerical epsilon floor (e.g., ε = 0.001 seconds) to all recorded times before computing ratios:import numpy as np import pandas as pd raw_data = { 'Solver_A': [0.12, 0.45, np.inf, 0.05, 1.20, 3.40, 0.85, np.inf], 'Solver_B': [0.08, 0.90, 2.10, 0.04, 0.95, np.inf, 1.10, 4.50], 'Solver_C': [0.25, 0.40, 1.80, 0.12, 2.10, 4.10, 0.90, 5.20], } df = pd.DataFrame(raw_data, index=[f'Problem_{i+1}' for i in range(8)]) epsilon = 1e-3 clean_df = df.clip(lower=epsilon) - Step 3: Calculate optimal baselines and performance ratios
For each problem row, determine the minimum execution time achieved across all solvers. Divide each solver’s runtime by this problem minimum to compute r_{p,s}. Set failed runs to a finite penalty value r_M strictly greater than the maximum observed finite ratio:min_times = clean_df.min(axis=1) ratios = clean_df.divide(min_times, axis=0) finite_max = ratios.replace(np.inf, np.nan).max().max() penalty_ratio = finite_max * 2.0 ratios_filled = ratios.fillna(penalty_ratio).replace(np.inf, penalty_ratio) - Step 4: Compute empirical cumulative distribution points
Define a grid of evaluation factors τ spanning from 1.0 (or log2(τ) = 0.0) up to the penalty threshold. For each candidate solver and each grid point τ, calculate the fraction of benchmark problems where r_{p,s} ≤ τ:tau_grid = np.logspace(0, np.log2(penalty_ratio), num=200, base=2.0) n_problems = len(df) profiles = {} for solver in clean_df.columns: solver_ratios = ratios_filled[solver].values rho = [np.sum(solver_ratios <= tau) / n_problems for tau in tau_grid] profiles[solver] = rho - Step 5: Render step-style cumulative curves with Matplotlib
Plot the resulting curves using step-style lines (plt.step) against log2(τ). Annotate the vertical axis as the fraction of problems solved and the horizontal axis as the performance ratio factor:import matplotlib.pyplot as plt plt.figure(figsize=(8, 5)) log2_tau = np.log2(tau_grid) for solver, rho in profiles.items(): plt.step(log2_tau, rho, label=solver, where='post', linewidth=2.0) plt.xlabel('log2(tau) (Performance Factor)', fontsize=12) plt.ylabel('rho_s(tau) (Fraction of Solved Problems)', fontsize=12) plt.title('Dolan-Moré Performance Profiles', fontsize=14) plt.ylim(0, 1.05) plt.xlim(0, np.log2(penalty_ratio)) plt.grid(True, linestyle='--', alpha=0.6) plt.legend(loc='lower right', frameon=True) plt.tight_layout() plt.savefig('performance_profile.png', dpi=300)
Output and how to read it
Interpreting a Dolan-Moré performance profile graph requires examining two distinct geometric properties of each solver’s trajectory: the vertical intercept on the left axis and the asymptotic plateau on the right side of the plot.
1. Efficiency (The Y-Intercept at τ = 1): The value of ρ_s(1)—where log2(τ) = 0 on the vertical axis—indicates the probability that solver s was the fastest or most efficient solver in the benchmark. If Solver A has ρ_A(1) = 0.65 while Solver B has ρ_B(1) = 0.30, Solver A solved 65% of the test suite faster than any competing algorithm, demonstrating computational superiority on easy-to-moderate problems.
2. Robustness (The Asymptotic Plateau as τ → ∞): As the factor τ increases toward the right edge of the plot, the curve ρ_s(τ) represents the fraction of problems that solver s can solve given an increasingly generous computational budget. The height of the curve at the maximum finite τ value indicates the solver’s overall robustness. If Solver B reaches ρ_B(τ) = 0.98 while Solver A plateaus at ρ_A(τ) = 0.75, Solver B successfully converged on 98% of the test suite, whereas Solver A failed on 25% of the problems.
3. Trajectory Dominance and Curve Crossing: When one solver’s curve remains strictly above another solver’s curve across all values of τ, the upper solver is unambiguously superior in both speed and reliability. However, curves frequently cross. A common pattern in numerical optimization is that aggressive second-order solvers exhibit high efficiency but lower robustness on ill-conditioned problems, whereas conservative, regularized solvers exhibit lower initial efficiency but higher asymptotic robustness.
Troubleshooting
When computing performance profiles on complex computational benchmarks, researchers frequently encounter numerical and methodological pitfalls. The following diagnostic solutions address the most common issues:
- Microsecond Run Times and Zero Division: When benchmarking fast solvers or trivial problem instances, runtimes can approach zero seconds, creating undefined division. Enforce an explicit measurement threshold ε (typically between 10^{-4} and 10^{-3} seconds) by applying
max(t_{p,s}, ε)to every cell before ratio computation. - Distorted Scaling from High Penalty Values: If the failure ratio r_M is set to an arbitrary massive value, the horizontal axis expands excessively, compressing meaningful competitive dynamics into an unreadable vertical sliver near zero. Set the plot boundary to a practical factor such as τ_max = 16 or 32, and plot solver failure plateaus at the right edge of that window.
- Inconsistent Termination Criteria: Comparing solvers that use different convergence tolerances or stopping criteria invalidates the benchmark. Ensure all solvers enforce identical feasibility and optimality thresholds before recording metrics.
- Small Sample Size Artifacts: If a benchmark suite contains fewer than 30 problem instances, the empirical CDF appears as a coarse staircase, making statistical inferences unreliable. Maintain at least 50 to 100 heterogeneous test instances to ensure generalizable results.
- The Elimination Paradox (Ranking Sensitivity): As demonstrated by Nicholas Gould and Jennifer Scott (2016), performance profiles can exhibit ranking sensitivity when solvers are added to or removed from the comparison set S. Because the denominator min_{s’} t_{p,s’} depends on the composition of S, removing an exceptionally fast solver can alter the relative performance ratios between the remaining solvers. Perform sensitivity analysis by computing profiles across subsets of solvers and verifying that relative rankings remain stable.
Free alternatives
Researchers can select from multiple mature, open-source software libraries to automate performance profile generation and benchmark reporting. The table below compares the primary free tools available across programming languages.
| Tool Name | Language / Ecosystem | Input Format | Supported Profiles | Output Formats | License |
|---|---|---|---|---|---|
| OptiProfiler | Python (PyPI) | NumPy arrays, CSV, Python callables | Performance, Data, Log-ratio | PNG, PDF, SVG (Matplotlib) | BSD-3-Clause |
| perprof-py | Python (PyPI, GitHub) | CSV, Pandas DataFrames | Performance profiles (Dolan-Moré) | PNG, PGF/TikZ (LaTeX), Bokeh HTML | MIT |
| BenchmarkProfiles.jl | Julia (Pkg) | Julia Matrix{Float64} | Performance and Data profiles | Plots.jl backends (PNG, PDF, SVG) | MIT |
| perfprof.m | MATLAB / GNU Octave | 2D MATLAB matrix | Performance profiles | MATLAB figure, EPS, PDF | Open Academic |
Choosing among these packages depends on your workflow. For Python workflows requiring seamless LaTeX integration in academic manuscripts, perprof-py is advantageous because it generates native PGF/TikZ vector graphics that match journal typography. For comprehensive optimization benchmarks requiring data profiles, OptiProfiler provides extensive analysis features and built-in budget scaling.
Advanced considerations: Data profiles and ranking sensitivity
While standard Dolan-Moré performance profiles are the gold standard for execution-time benchmarks, derivative-free optimization (DFO) and simulation-based engineering present unique challenges. In these domains, evaluating the objective function requires running expensive numerical simulations, making function evaluations the dominant computational cost rather than wall-clock solver overhead.
To address this domain, Jorge Moré and Stefan Wild (2009) introduced data profiles. A data profile measures the fraction of problems that a derivative-free algorithm can solve within a specific budget of function evaluations. To ensure scale-invariance across problems of different dimensionalities, the evaluation budget is expressed in terms of simplex gradients: κ = k / (n_p + 1), where k is the number of function evaluations performed and n_p is the number of decision variables in problem p. Data profiles allow researchers to assess which algorithms make rapid progress under tight computational budgets, complementing the asymptotic view provided by standard performance profiles.
Furthermore, methodological rigor demands awareness of the Gould-Scott critique (2016). When publishing comparative solver studies, researchers should not rely exclusively on a single performance profile plot. Best practices include: (1) publishing the raw, unnormalized performance matrix in supplementary materials; (2) reporting the performance profile alongside traditional tabular summaries of median runtimes; and (3) generating successive profiles that systematically exclude dominant solvers to verify that secondary solver rankings do not suffer from denominator distortion.
The data table below compares benchmark performance profile characteristics across key numerical optimization solvers and profiling toolkits, detailing efficiency intercepts, robustness thresholds, and implementation specifications.
Frequently asked questions
What is the difference between efficiency and robustness in a performance profile?
Efficiency measures how often a solver is the fastest or most resource-efficient algorithm across a test suite, represented by the profile’s vertical intercept at tau = 1 (or log2(tau) = 0). Robustness measures the solver’s overall reliability in successfully solving problems regardless of speed, represented by the asymptotic height of the profile curve as tau approaches infinity or the maximum evaluated factor.
How should solver failures and timeouts be handled in performance profiles?
Solver failures, timeouts, and divergence must be assigned a performance ratio greater than or equal to the maximum ratio achieved on any successfully solved problem, denoted as r_M, or set to infinity. In cumulative distribution calculations, failures are placed beyond the maximum evaluated tau factor so they never contribute to the solved percentage within the visible plotting window.
Why are performance profiles commonly plotted on a logarithmic scale?
Performance ratios naturally span several orders of magnitude, ranging from 1 (the best solver) to hundreds or thousands for slow runs. Plotting against log2(tau) ensures that proportional performance differences—such as a solver taking twice or four times as long as the winner—receive equal visual spacing along the horizontal axis, preventing extreme outliers from compressing the informative region near tau = 1.
What is the Gould and Scott pitfall in performance profiles?
Nicholas Gould and Jennifer Scott demonstrated in 2016 that because the performance ratio normalizes by the minimum runtime across the solver set, adding or removing a solver changes the baseline denominator for individual problems. In certain configurations, this can alter or invert the relative ordering between two other solvers. Researchers should present successive profiles or sensitivity checks when evaluating closely matched algorithms.
When should researchers use data profiles instead of performance profiles?
Data profiles, introduced by Jorge More and Stefan Wild in 2009, are preferred when benchmarking derivative-free or simulation-based optimization algorithms with strictly limited evaluation budgets. While performance profiles measure relative speed against competing solvers, data profiles track the fraction of problems solved as a function of the number of function evaluations scaled by problem dimension.
Can performance profiles be used for metrics other than CPU runtime?
Yes. Performance profiles can evaluate any positive, non-zero performance metric where smaller values represent superior performance, including iteration counts, objective function evaluations, gradient calculations, memory consumption, or communication rounds in distributed optimization algorithms.
Sources
- Dolan, E. D., & Moré, J. J. (2002). Benchmarking optimization software with performance profiles. Mathematical Programming, 91(3), 201–213. https://link.springer.com/article/10.1007/s101070100263 (Accessed 2026-11-14).
- Dolan, E. D., & Moré, J. J. (2001). Benchmarking Optimization Software with Performance Profiles (Preprint cs/0102001). arXiv. https://arxiv.org/abs/cs/0102001 (Accessed 2026-11-14).
- Gould, N. I. M., & Scott, J. A. (2016). A note on performance profiles for benchmarking software. ACM Transactions on Mathematical Software, 43(2), Article 15. https://epubs.stfc.ac.uk (Accessed 2026-11-14).
- Moré, J. J., & Wild, S. M. (2009). Benchmarking derivative-free optimization algorithms. SIAM Journal on Optimization, 20(1), 172–191. https://doi.org/10.1137/080724083 (Accessed 2026-11-14).
- Siqueira, A. S., Toledo, F. M. B., & Santos, M. N. (2016). perprof-py: A Python Package for Performance Profiles of Mathematical Optimization Software. Journal of Open Research Software. https://github.com/abelsiqueira/perprof-py (Accessed 2026-11-14).
- Ragonneau, T. M., & Zhang, Z. (2024). OptiProfiler: A Python package for benchmarking derivative-free and derivative-based optimization solvers. https://www.optprof.com/latest/ (Accessed 2026-11-14).
- JuliaSmoothOptimizers. (2024). BenchmarkProfiles.jl Documentation. https://jso.dev/BenchmarkProfiles.jl/stable/ (Accessed 2026-11-14).
Data: Optimization Solver Benchmarking Performance Profiles and Metrics
Comparative evaluation of mathematical optimization solvers and benchmarking software toolkits, compiled from published literature including Dolan and More (2002), Gould and Scott (2016), and official software documentation.
Period covered: 2002/2026. Coverage: Global.
| Solver or Toolkit | Ecosystem | Problem Class | Efficiency Intercept (%) | Robustness Plateau (%) | Failure Penalty Handling | Open Source License |
|---|---|---|---|---|---|---|
| MINOS | Fortran / C | General nonlinear programming | 38 | 72 | Infinite ratio r_M | Academic Open Source |
| SNOPT | Fortran / C / Python | Sequential quadratic programming (SQP) | 45 | 84 | Infinite ratio r_M | Academic / Commercial |
| LANCELOT | Fortran / C | Augmented Lagrangian NLP | 28 | 78 | Infinite ratio r_M | Academic Open Source |
| KNITRO | C / C++ / Python | Interior point and active set NLP | 52 | 91 | Infinite ratio r_M | Commercial |
| Ipopt | C++ / Python / Julia | Interior point nonlinear optimization | 42 | 88 | Infinite ratio r_M | EPL-2.0 |
| HiGHS | C++ / Python / Julia | Linear and mixed-integer programming | 48 | 93 | Infinite ratio r_M | MIT |
| OptiProfiler | Python | Benchmarking and profile generation | 100 | 100 | Customizable penalty threshold | BSD-3-Clause |
| perprof-py | Python | Dolan-More profile generation | 100 | 100 | Infinite ratio with cutoff | MIT |
| BenchmarkProfiles.jl | Julia | Dolan-More and data profile generation | 100 | 100 | Infinite ratio with NaN masking | MIT |
Sources
- Dolan & More (2002) Benchmarking Optimization Software with Performance Profiles (accessed 2026-11-14)
- Gould & Scott (2016) A note on performance profiles for benchmarking software (accessed 2026-11-14)
- perprof-py: A Python Package for Performance Profiles (accessed 2026-11-14)
- OptiProfiler Documentation and Solver Benchmarks (accessed 2026-11-14)
- BenchmarkProfiles.jl Official Documentation (accessed 2026-11-14)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







