CTest: CMake Test Driver for Research

CTest is the test driver program distributed natively with Kitware‘s CMake build system. For research software engineers, computational scientists, and academic developers working on simulation engines, numerical libraries, and data processing pipelines, CTest provides a standardized, language-agnostic test harness. This guide explains how to configure CTest in CMake, register test executables, apply execution timeouts and label filters, execute test suites in parallel, and diagnose failures in automated continuous integration workflows.

What it is used for in research

Computational science and research software engineering require rigorous verification to ensure computational reproducibility and algorithmic correctness. Scientific software—ranging from astrophysical fluid dynamics simulators to high-throughput bioinformatics pipelines—frequently integrates compiled code written in C, C++, and Fortran with high-level scripting languages. CTest acts as an overarching test runner that orchestrates and executes test suites across diverse research computing environments, from local developer workstations to high-performance computing (HPC) clusters.

In academic research, CTest is commonly used for:

• Unit testing and regression verification: Coordinating unit test execution with frameworks such as Catch2 and GoogleTest to prevent numerical and logical regressions during active software development.
• Algorithmic validation against reference datasets: Executing simulation binaries against published benchmark datasets and comparing numerical output to established analytical solutions within specified floating-point tolerances.
• Continuous integration (CI) automation: Powering automated testing matrices within containerized environments (such as Docker vs virtual machine setups) and cloud pipelines (GitHub Actions, GitLab CI) to validate pull requests before code merges.
• Compliance with journal software standards: Meeting peer-review testing criteria established by computational journals, including the Journal of Open Source Software (JOSS), which mandate automated, reproducible test suites for accepted software papers.
• Multi-configuration and multi-platform testing: Ensuring research code compiles and passes tests across multiple compiler toolchains (GCC, Clang, Intel oneAPI, MSVC) and execution configurations (Debug, Release, RelWithDebInfo).

Install or access (official source only)

CTest is an integrated component of CMake, maintained and published by Kitware, Inc. It does not require a separate installation package, standalone license, or proprietary subscription. Installing CMake on your operating system automatically installs the corresponding ctest executable into the system binary path.

Official installation methods for research environments include:

• Ubuntu and Debian Linux: Install via the system package manager:

  sudo apt-get update && sudo apt-get install -y cmake

  For high-performance systems requiring bleeding-edge CMake releases, Kitware provides an official APT repository at apt.kitware.com.

• Red Hat Enterprise Linux, Rocky Linux, and Fedora: Install via dnf:

  sudo dnf install -y cmake

• macOS: Install via Homebrew:

  brew install cmake

• Python Environments (Virtualenv / Conda): Official pre-compiled CMake and CTest binaries are distributed by Kitware on PyPI:

  pip install cmake

• Windows: Download the official binary MSI installer or ZIP package directly from the Kitware CMake download portal at https://cmake.org/download/.

After installation, verify that the ctest executable is available and matches your CMake version by executing:

ctest --version

Step-by-step task (numbered, with menu names or commands exactly as in the current docs)

Follow these numbered steps to configure, build, and execute automated test suites using CMake and CTest in a research software repository.

Step 1: Enable testing in the root CMake configuration

Open the top-level CMakeLists.txt file in your project repository. To activate CTest functionality across your project, add the include(CTest) module command or the enable_testing() command. The recommended approach in modern CMake is include(CTest), which automatically defines the standard BUILD_TESTING cache variable (defaulting to ON) and invokes enable_testing():

cmake_minimum_required(VERSION 3.20)
project(ScientificSimulation VERSION 1.0.0 LANGUAGES CXX)

# Standard CTest integration
include(CTest)

if(BUILD_TESTING)
  add_subdirectory(tests)
endif()

Step 2: Register test executables with add_test

In your tests/CMakeLists.txt file, define the test executable and register it with CTest using the official add_test() command. CTest evaluates tests by monitoring process exit status: a process return code of 0 indicates success, while any non-zero return code or fatal signal indicates failure.

# Define the test binary
add_executable(test_matrix_solver test_matrix_solver.cpp)
target_link_libraries(test_matrix_solver PRIVATE ScientificLib)

# Register the test case with CTest
add_test(
  NAME MatrixSolverConvergence
  COMMAND test_matrix_solver --tolerance 1e-8 --iterations 500
)

Step 3: Configure test properties and timeouts

Scientific simulations can stall or enter infinite loops if convergence criteria fail. Use set_tests_properties() to configure execution limits, categorization labels, and expected outputs:

# Apply execution timeout and categorization labels
set_tests_properties(MatrixSolverConvergence PROPERTIES
  TIMEOUT 60
  LABELS "unit;numerical"
  PASS_REGULAR_EXPRESSION "CONVERGENCE_ACHIEVED"
  FAIL_REGULAR_EXPRESSION "NAN_DETECTED;DIVERGENCE"
)

# Register a test expected to fail on invalid boundary conditions
add_test(
  NAME MatrixSolverInvalidDimensions
  COMMAND test_matrix_solver --rows -1
)
set_tests_properties(MatrixSolverInvalidDimensions PROPERTIES
  WILL_FAIL TRUE
  LABELS "unit;validation"
)

Step 4: Build the project and run CTest

Generate the build system, compile the test binaries, and execute CTest from the root of your project using the --test-dir argument (available in CMake 3.20 and later):

# Configure and build
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

# Execute all registered tests
ctest --test-dir build --output-on-failure

The --output-on-failure flag ensures that if any test fails, its standard output and standard error are printed directly to the terminal, avoiding the need to manually open log files.

Step 5: Filter and parallelize test runs

CTest provides command-line flags to filter test execution by regular expression, label, or prior failure state, and to accelerate execution across multi-core CPUs:

• Parallel execution: Run tests concurrently across 8 worker processes:

  ctest --test-dir build -j 8

• Filter by test name regex (-R): Run only tests matching a name pattern:

  ctest --test-dir build -R "MatrixSolver.*"

• Exclude tests by regex (-E): Exclude resource-intensive benchmarks:

  ctest --test-dir build -E "Benchmark"

• Filter by label (-L): Execute only fast unit tests:

  ctest --test-dir build -L "unit"

• Re-run only failed tests: Restrict execution to tests that failed during the preceding invocation:

  ctest --test-dir build --rerun-failed --output-on-failure

• Flaky test validation (--repeat): Repeat tests to ensure stability under stochastic conditions:

  ctest --test-dir build --repeat until-pass:3

Output and how to read it

When CTest executes, it displays a concise summary for each test case indicating the test index, name, execution outcome, and duration in seconds:

Test project /home/researcher/ScientificSimulation/build
    Start 1: MatrixSolverConvergence
1/2 Test #1: MatrixSolverConvergence ...........   Passed    0.14 sec
    Start 2: MatrixSolverInvalidDimensions
2/2 Test #2: MatrixSolverInvalidDimensions .....   Passed    0.02 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.17 sec

Understanding the key diagnostic fields:

• Passed: The executable exited with return code 0 (or exited with non-zero if the WILL_FAIL property was set to TRUE), and any defined PASS_REGULAR_EXPRESSION matched while no FAIL_REGULAR_EXPRESSION matched.
• ***Failed: The executable returned a non-zero exit code or failed to satisfy regular expression checks. CTest outputs the process return code (e.g. Exit code 0x1).
• ***Timeout: The test process exceeded the duration specified by the TIMEOUT property (or the global CLI --timeout limit) and was terminated by CTest.
• ***Exception: Other: Indicates an unhandled operating system exception or fatal signal, such as a segmentation fault (SIGSEGV) or floating-point exception (SIGFPE).
• Log file artifacts: Comprehensive terminal logs, timestamps, and environment variable states for all executed tests are recorded in build/Testing/Temporary/LastTest.log. A list of failed test names is saved in build/Testing/Temporary/LastTestsFailed.log.

Troubleshooting

Below are common operational issues encountered when executing CTest in research software environments and their verified resolutions:

• “No test configuration file found!”: This error occurs when ctest is invoked from a directory that does not contain a CTestTestfile.cmake file. To resolve this, specify the build directory using ctest --test-dir <path_to_build> or execute ctest from inside the build folder. Ensure that enable_testing() or include(CTest) is called in the root CMakeLists.txt before any add_subdirectory() calls.
• Missing test binaries with multi-configuration generators: On multi-configuration generators (such as Visual Studio or Xcode), CTest does not know which build configuration to execute by default. Pass the -C or --build-config flag:

  ctest --test-dir build -C Release

• Tests timing out during intensive calculations: Long-running numerical benchmarks may exceed default CI limits. Increase the per-test timeout in CMakeLists.txt using set_tests_properties(<test_name> PROPERTIES TIMEOUT 300) or override all timeouts on the command line using ctest --timeout 300.
• Shared library loading failures (error while loading shared libraries): If test binaries fail because custom shared libraries built by the project are not found in the dynamic linker path, inject the build directory into the test environment using:

  set_tests_properties(<test_name> PROPERTIES ENVIRONMENT "LD_LIBRARY_PATH=${CMAKE_BINARY_DIR}/lib:$ENV{LD_LIBRARY_PATH}")

• Zero tests discovered: If ctest -N reports zero tests, confirm that BUILD_TESTING was not set to OFF during the CMake configuration step (cmake -B build -DBUILD_TESTING=ON) and that enable_testing() appears in the root CMakeLists.txt.

Free alternatives (table)

While CTest is the standard test runner for CMake-managed projects in research software engineering, several free and open-source testing tools and drivers serve related scientific computing ecosystems:

Tool	Primary Ecosystem	Integration Model	Parallel Execution	Scientific Computing Strengths
CTest	C, C++, Fortran, Multi-language	Native CMake test driver	Built-in (-j <jobs>)	Standard across compiled scientific simulation codes, HPC modules, and multi-language builds.
pytest	Python	Standalone CLI test framework	Via pytest-xdist (-n auto)	Dominant in scientific data analysis, machine learning workflows, and NumPy/SciPy testing.
Cargo test	Rust	Native Cargo build/test runner	Built-in parallel threads	Rapidly growing in high-performance research tools, bio-informatics parsers, and systems software.
GNU Autotest	C, C++, Shell	GNU Autotools harness	Supported via parallel test harnesses	Common in legacy Unix scientific utilities and long-standing academic packages.
Catch2	C++	Test framework with CTest adapter	Driven via CTest parallel jobs	Expressive BDD assertions and floating-point matchers for research C++ libraries.

FAQ

What is the difference between ctest and make test (or ninja test)?

The build tool targets (make test or ninja test) invoke CTest behind the scenes, but they suppress individual test command-line arguments and buffering options. Running ctest directly allows you to pass essential flags such as --output-on-failure, regex filters (-R), label selectors (-L), and custom worker process counts (-j).

How does CTest determine whether a test passed or failed?

CTest evaluates tests primarily by checking the process exit status: a return code of 0 signifies success, while any non-zero exit code or fatal signal represents failure. If the WILL_FAIL property is set to TRUE, CTest inverts this evaluation so that non-zero exit codes pass. Additionally, tests fail if custom regular expressions defined in FAIL_REGULAR_EXPRESSION match the console output, or if PASS_REGULAR_EXPRESSION is specified and fails to match.

Can CTest execute test suites written in Python or Fortran?

Yes. CTest is language-agnostic because it executes any binary executable, shell script, or interpreter command registered via add_test(NAME <name> COMMAND <command> ...). Research projects routinely use CTest to drive compiled Fortran executables, Python validation scripts (python3 test_script.py), and shell benchmarks.

How do you view console output only when a test fails?

Pass the --output-on-failure flag when invoking CTest (ctest --test-dir build --output-on-failure), or set the environment variable CTEST_OUTPUT_ON_FAILURE=1 in your shell or CI workflow. This ensures passing tests produce a clean summary while failing tests output full standard error and diagnostic logs.

What is the difference between include(CTest) and enable_testing()?

The enable_testing() command solely enables the test registration mechanism in the current CMake directory and its subdirectories. The include(CTest) module command wraps enable_testing(), adds the standard user-configurable BUILD_TESTING cache option, and configures automated dashboard submission targets for CDash server reporting.

Sources (official docs)

• Kitware, Inc. “ctest(1) — CMake 3.31 Documentation.” Kitware Official Documentation. https://cmake.org/cmake/help/latest/manual/ctest.1.html (Accessed November 9, 2026).
• Kitware, Inc. “add_test — CMake 3.31 Documentation.” Kitware Official Documentation. https://cmake.org/cmake/help/latest/command/add_test.html (Accessed November 9, 2026).
• Kitware, Inc. “enable_testing — CMake 3.31 Documentation.” Kitware Official Documentation. https://cmake.org/cmake/help/latest/command/enable_testing.html (Accessed November 9, 2026).
• Kitware, Inc. “CTest — CMake 3.31 Documentation.” Kitware Official Documentation. https://cmake.org/cmake/help/latest/module/CTest.html (Accessed November 9, 2026).
• Better Scientific Software (BSSw.io) / Exascale Computing Project. “How to Improve Testing for CSE Software.” https://bssw.io/items/how-to-improve-testing-for-cse-software (Accessed November 9, 2026).