Skip to main content
v2026.11,986 entries · CC-BY 4.0

Agisoft Metashape: Formulas, RMSE, and Methods

Learn how to calculate GSD, evaluate reprojection RMSE, and report Agisoft Metashape photogrammetric models in archaeological research with worked examples.

Written and maintained by CASRAI Editorial Board

Published

On this page

What is Agisoft Metashape? Definition and photogrammetric scope

Agisoft Metashape is a commercial photogrammetry suite that reconstructs 3D spatial geometry from overlapping 2D digital imagery using Structure-from-Motion (SfM) and Multi-View Stereo (MVS) algorithms. Developed by Agisoft LLC, it serves archaeological field surveys, architectural recording, and spatial analysis by automating feature detection, bundle adjustment, and geodetic coordinate transformations (Bedford, 2017; Luhmann et al., 2019). By aligning photographic rays without requiring pre-calibrated lenses, Metashape generates metric point clouds, textured meshes, digital elevation models, and orthomosaics tied to coordinate reference frames standardized by the Open Geospatial Consortium.

Photogrammetric formulas and mathematical framework

Structure-from-Motion photogrammetry reconstructs scene geometry through projective transformations evaluated via Ground Sampling Distance (GSD) and Root Mean Square Reprojection Error (RMSE).

Ground Sampling Distance (GSD)

Ground Sampling Distance defines surface distance between adjacent pixel centers on the ground:

$$text{GSD} = frac{H times s_w}{f times W_{text{px}}}$$

Where $H$ is standoff distance in mm, $s_w$ is sensor width in mm, $f$ is focal length in mm, and $W_{text{px}}$ is sensor width in pixels.

Reprojection Error and Root Mean Square Error (RMSE)

Bundle adjustment minimizes Euclidean distance between detected image features $(x_i, y_i)$ and back-projected coordinates $(x’_i, y’_i)$:

$$d_i = sqrt{(x’_i – x_i)^2 + (y’_i – y_i)^2}$$

Across all $N$ observed tie point projections, precision is evaluated as Root Mean Square Error:

$$text{RMSE}_{text{reproj}} = sqrt{frac{1}{N} sum_{i=1}^{N} left[(x’_i – x_i)^2 + (y’_i – y_i)^2right]}$$

Where $N$ is total projections, $(x_i, y_i)$ are detected 2D coordinates in pixels, and $(x’_i, y’_i)$ are ray projections through estimated camera parameters.

3D Ground Control Point (GCP) Error

Georeferencing accuracy is determined by comparing surveyed coordinates against estimated positions across $M$ targets:

$$text{RMSE}_{text{GCP}} = sqrt{frac{1}{M} sum_{j=1}^{M} left[(X_{text{est}, j} – X_{text{obs}, j})^2 + (Y_{text{est}, j} – Y_{text{obs}, j})^2 + (Z_{text{est}, j} – Z_{text{obs}, j})^2right]}$$

Worked example: calculating GSD and GCP reprojection RMSE in excavation survey

Consider a 5 m × 5 m archaeological excavation trench recorded with a pole-mounted camera. An analyst calculates the Ground Sampling Distance and computes 3D GCP error across four surveyed control markers.

Survey Parameters

  • Sensor width ($s_w$): 36.0 mm (full-frame sensor).
  • Image width ($W_{text{px}}$): 6,000 pixels.
  • Focal length ($f$): 24.0 mm.
  • Standoff distance ($H$): 8.0 m (8,000 mm).

Step 1: Ground Sampling Distance

$$text{GSD} = frac{8{,}000text{ mm} times 36.0text{ mm}}{24.0text{ mm} times 6{,}000text{ px}} = frac{288{,}000}{144{,}000} = 2.0text{ mm/pixel}$$

The survey achieves a Ground Sampling Distance of 2.0 mm per pixel.

Step 2: 3D GCP Residuals

Four total station surveyed targets exhibit the following spatial residuals in millimeters:

  • GCP 1: $Delta X = +0.20$, $Delta Y = -0.10$, $Delta Z = +0.20text{ mm} implies d_1^2 = 0.0400 + 0.0100 + 0.0400 = 0.0900text{ mm}^2$
  • GCP 2: $Delta X = -0.10$, $Delta Y = +0.30$, $Delta Z = -0.20text{ mm} implies d_2^2 = 0.0100 + 0.0900 + 0.0400 = 0.1400text{ mm}^2$
  • GCP 3: $Delta X = +0.30$, $Delta Y = -0.20$, $Delta Z = -0.10text{ mm} implies d_3^2 = 0.0900 + 0.0400 + 0.0100 = 0.1400text{ mm}^2$
  • GCP 4: $Delta X = -0.20$, $Delta Y = +0.10$, $Delta Z = +0.30text{ mm} implies d_4^2 = 0.0400 + 0.0100 + 0.0900 = 0.1400text{ mm}^2$

Step 3: Calculating Spatial RMSE

$$text{RMSE}_{text{GCP}} = sqrt{frac{0.0900 + 0.1400 + 0.1400 + 0.1400}{4}} = sqrt{0.1275} approx 0.3571text{ mm} approx 0.36text{ mm}$$

Step 4: Tie Point Reprojection Error

For a tie point observed across four photos with residuals $(e_x, e_y)$ of $(+0.25, -0.15)$, $(-0.30, +0.20)$, $(+0.15, +0.10)$, and $(-0.20, -0.25)$ px:

$$text{RMSE}_{text{reproj}} = sqrt{frac{0.0850 + 0.1300 + 0.0325 + 0.1025}{4}} = sqrt{0.0875} approx 0.2958text{ px} approx 0.30text{ px}$$

Interpretation and reading guidance

An RMSE of 0.36 mm on control points with a 2.0 mm GSD confirms sub-millimeter georeferencing accuracy, surpassing standard 1 cm excavation tolerances. The reprojection error of 0.30 pixels confirms optical alignment below the sub-pixel threshold.

Diagnostic Metric Optimal Range Acceptable Threshold Diagnostic Interpretation
Reprojection Error < 0.50 px 0.50 – 1.00 px Sub-pixel alignment confirms accurate lens model. Errors > 1.0 px require gradual selection tie-point filtering.
Tie Point Scale Error < 0.25 units 0.25 – 0.40 units Dimensionless unit in bundle adjustment. Values > 0.50 indicate scale-space feature mismatches.
GCP 3D Residuals < 1.0 × GSD 1.0 – 2.0 × GSD Model conforms tightly to survey grid. Values > 3.0 × GSD point to coordinate typos or marker misplacement.
Check Point Residuals < 1.5 × GSD 1.5 – 3.0 × GSD Independent validation. Discrepancies between GCP and check point errors indicate non-linear doming.
Reconstruction Uncertainty < 10.0 10.0 – 15.0 Reflects ray convergence angle. Values > 20 indicate shallow baselines that weaken depth triangulation.

Assumptions and common mistakes in photogrammetric recording

Structure-from-Motion photogrammetry relies on specific optical conditions. Deviating from these principles introduces geometric distortion.

Core Methodological Assumptions

  • Static Scene Geometry: Targets, terrain, and excavation surfaces must remain motionless throughout capture. Windblown foliage or disturbed soil compromises tie point matching.
  • Diffuse Lambertian Reflectance: Photogrammetric algorithms assume surfaces scatter light equally in all directions. Wet or specular artifacts create angle-dependent highlights that confuse feature matching.
  • Sufficient Visual Texture: Matching requires distinct high-frequency surface patterns. Uniform textures, such as smooth concrete or monochrome sand, yield sparse point clouds.
  • Stable Internal Calibration: Focal length and lens geometry must remain constant within an image group. Autofocus must be locked to prevent focal drift between frames.

Common Operational Mistakes

  • Overparameterizing Calibration: Enabling advanced distortion parameters (k4, p3, p4) on flat datasets without oblique images causes non-linear doming deformation across flat excavation trenches.
  • Omitting Independent Check Points: Assigning all markers as Ground Control Points conceals model warping. At least 25% of surveyed markers should be designated as check points to verify true accuracy.
  • Inadequate Photo Overlap: Capturing less than 60% sidelap and 75% forward overlap leads to alignment splits and holes in the dense surface reconstruction.
  • Harsh Cast Shadows: Direct midday sunlight creates moving shadows that corrupt texture mapping. Overcast conditions provide optimal diffuse illumination.

How to report photogrammetric data in research (APA example)

Archaeological documentation requires transparent reporting of hardware parameters, software versions, error budgets, and coordinate reference frames per Historic England guidelines (Bedford, 2017) and FAIR data principles.

Sample APA 7th Reporting Paragraph

Archaeological surface documentation was conducted via close-range digital photogrammetry processed in Agisoft Metashape Professional (Version 2.1; Agisoft LLC, 2024). A dataset of 142 digital photographs was captured using a 61.0-megapixel full-frame digital camera (Sony α7R IV) equipped with a 24.0 mm prime lens locked at fixed focus (ISO 100, f/8.0, 1/250 s) under diffuse overcast lighting. Flight passes achieved an average 75% forward and 65% lateral overlap, yielding a mean Ground Sampling Distance (GSD) of 2.0 mm/pixel at an 8.0 m standoff distance. Georeferencing was established with eight targets surveyed using an electronic total station (precision ± 1.0 mm), with four targets designated as Ground Control Points (GCPs) and four withheld as independent check points. The bundle adjustment converged across 248,310 tie points with an overall Root Mean Square Error (RMSE) of 0.30 pixels. Total 3D spatial RMSE was 0.36 mm on control points and 0.49 mm on check points. Orthomosaics and digital surface models were exported in the British National Grid (OSGB36 / EPSG:27700) coordinate system.

APA 7th Reference Citation

Agisoft LLC. (2024). Agisoft Metashape Professional Edition (Version 2.1) [Computer software]. Agisoft LLC. https://www.agisoft.com/downloads/user-manuals/

How to calculate photogrammetric metrics in Excel, R, and Python

Survey teams can verify Ground Sampling Distance and spatial error metrics using standard computational scripts before running full reconstruction pipelines.

Excel Calculation Syntax

Enter distance $H$ (mm) in A2, sensor width $s_w$ (mm) in B2, focal length $f$ (mm) in C2, and image width $W_{text{px}}$ (px) in D2 to compute GSD (mm/px):

=(A2 * B2) / (C2 * D2)

To compute 3D GCP RMSE from residual columns B (ΔX), C (ΔY), and D (ΔZ) across rows 2 through 5:

=SQRT(SUMSQ(B2:D5) / COUNT(B2:B5))

R Implementation

calc_gsd <- function(H_mm, sensor_w_mm, focal_mm, image_w_px) {
  (H_mm * sensor_w_mm) / (focal_mm * image_w_px)
}

calc_gcp_rmse <- function(residuals_matrix) {
  sqrt(mean(rowSums(residuals_matrix^2)))
}

gsd <- calc_gsd(8000, 36.0, 24.0, 6000)
gcp_matrix <- matrix(
  c(0.20, -0.10, 0.20, -0.10, 0.30, -0.20, 0.30, -0.20, -0.10, -0.20, 0.10, 0.30),
  ncol = 3, byrow = TRUE
)
cat(sprintf("GSD: %.2f mm/pixel | 3D GCP RMSE: %.4f mmn", gsd, calc_gcp_rmse(gcp_matrix)))

Python Implementation

Standalone Python code computing GSD and reprojection RMSE without external libraries, suitable for preprocessing alongside Rasterio:

import math

def calculate_gsd(standoff_mm: float, sensor_w_mm: float, focal_mm: float, image_w_px: int) -> float:
    """Calculate Ground Sampling Distance (GSD) in mm/pixel."""
    return (standoff_mm * sensor_w_mm) / (focal_mm * image_w_px)

def calculate_reprojection_rmse(residuals_xy: list[tuple[float, float]]) -> float:
    """Calculate Root Mean Square Reprojection Error in pixels."""
    sq_err = [dx**2 + dy**2 for dx, dy in residuals_xy]
    return math.sqrt(sum(sq_err) / len(sq_err))

gsd_val = calculate_gsd(8000.0, 36.0, 24.0, 6000)
reproj_errs = [(0.25, -0.15), (-0.30, 0.20), (0.15, 0.10), (-0.20, -0.25)]
print(f"GSD: {gsd_val:.2f} mm/pixel | Reprojection RMSE: {calculate_reprojection_rmse(reproj_errs):.4f} pixels")

Frequently asked questions

What is the difference between Agisoft Metashape Standard and Professional?

Metashape Standard handles basic Structure-from-Motion alignment, dense clouds, and textured 3D meshes without georeferencing. Metashape Professional adds Ground Control Point support, coordinate system transformations, digital elevation model and orthomosaic export, and Python API scripting required for archaeological field surveys.

What is considered an acceptable reprojection error in Agisoft Metashape?

In close-range archaeological photogrammetry, root mean square reprojection error below 0.50 pixels represents optimal sub-pixel convergence. Values up to 1.00 pixel remain acceptable for landscape surveys. Reprojection errors exceeding 1.00 pixel indicate inaccurate keypoint matching or lens distortion, requiring gradual selection filtering before dense reconstruction.

Why should check points be separated from ground control points?

Ground Control Points (GCPs) actively constrain bundle adjustment equations, so their residuals measure optimization fit rather than true external accuracy. Designating 20% to 30% of surveyed markers as independent check points ensures they do not influence calibration, providing unbiased validation of real-world scale and geometry.

How does Ground Sampling Distance relate to 3D model accuracy?

Ground Sampling Distance (GSD) sets the upper resolution limit of photographic pixels on the ground. In a calibrated photogrammetric project, horizontal spatial accuracy typically achieves 1 to 2 times the GSD, while vertical height accuracy ranges between 1.5 and 3 times the GSD. Finer GSD resolves smaller features but requires closer camera standoffs or higher sensor resolution.

Sources

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Agisoft Metashape: Formulas, RMSE, and Methods

Ask your first 2 questions free below. Subscribers get 150 a day for $29 a month.

An AI assistant specialized in research administration. It cites the sources behind every answer, labels web answers and says when it can't answer.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →