Skip to main content
v2026.11,986 entries · CC-BY 4.0

Differentiable Rendering: Mitsuba 3, nvdiffrast and PyTorch3D

Differentiable rendering lets gradients flow from an image back to scene parameters. A Mitsuba 3 walkthrough, with nvdiffrast and PyTorch3D compared.

Written and maintained by CASRAI Editorial Board

Published

On this page

Differentiable rendering is rendering that can be differentiated: the image-formation step is treated as a function from a scene description (geometry, materials, lighting, camera) to an image, and software computes how the image changes when those scene parameters change. With those derivatives, a gradient-based optimizer can adjust the scene until its rendering matches a target image, which is how researchers solve inverse problems such as recovering a material colour, a pose or a shape from pictures. Three open tools cover most research use: Mitsuba 3, nvdiffrast and PyTorch3D.

What differentiable rendering is used for in research

The Mitsuba 3 documentation states the idea directly: Mitsuba 3 can be used to solve inverse problems involving light using a technique known as differentiable rendering. Rendering is written as a function f(x) that maps a scene description x to an image y. Differentiating that function gives dy/dx, and combined with a differentiable objective g(y) that scores how close the image is to a target, a gradient-based optimizer such as stochastic gradient descent or Adam updates the scene parameters step by step.

The Mitsuba 3 tutorial index shows the range of problems this method is applied to in practice: forward inverse rendering, caustics optimization, object pose estimation, volumetric inverse rendering, shape optimization, radiance field reconstruction in the style of NeRF, polarizer optimization, and interoperability with PyTorch. nvdiffrast describes itself as a PyTorch library of high-performance primitive operations for rasterization-based differentiable rendering, and PyTorch3D, whose stated goal is to accelerate research at the intersection of deep learning and 3D, includes a modular differentiable rendering API next to its other 3D operators.

Two families of tool

The tools split by how the image is produced. nvdiffrast is explicitly rasterization-based. Mitsuba 3 is a light-transport renderer whose gradient-capable integrators (the tutorial uses prb, Path Replay Backpropagation) combine a path tracer with an adjoint pass that produces gradients. PyTorch3D offers differentiable rendering as one module inside a broader 3D deep-learning library. The data table at the end of this guide sets the three side by side, and every cell comes from the projects’ own documentation.

How to install or access differentiable rendering tools

Mitsuba 3

Mitsuba 3 installs from PyPI, which its documentation calls the recommended method:

  1. pip install mitsuba

The install also brings in Dr.Jit if it is not already present. Building from the git source is described in the project’s developer guide. For differentiable rendering you must choose a variant whose name ends in _ad, such as cuda_ad_rgb or llvm_ad_rgb.

nvdiffrast

The nvdiffrast README gives these commands for installing from source:

  1. pip install setuptools wheel ninja
  2. pip install git+https://github.com/NVlabs/nvdiffrast.git --no-build-isolation

Its documentation is hosted at nvlabs.github.io/nvdiffrast. The code is released under the Nvidia Source Code License, so read the licence text before reusing it in a project, and note that the repository does not accept outside pull requests. See the guide to open source licences for how licence terms affect reuse.

PyTorch3D

The PyTorch3D INSTALL file lists a conda package for CUDA on Linux and a source build:

  1. conda install pytorch3d -c pytorch3d (Linux, CUDA)
  2. pip install "git+https://github.com/facebookresearch/pytorch3d.git" to build from the main branch; append @stable to the repository URL for the latest released version.

When building from source, CUDA support is compiled in if CUDA is available in your PyTorch installation, or if you set FORCE_CUDA=1. Pin the tool versions you used in an environment file, as described in sharing a reproducible conda environment.

How to recover a scene parameter with Mitsuba 3 step by step

The official gradient-based optimization tutorial renders a Cornell box as a reference image, deliberately changes the colour of the red wall, and then recovers the original colour by differentiating through the renderer. The steps below follow it.

  1. Import the libraries and select a differentiable variant.
    import drjit as drimport mitsuba as mimi.set_variant('llvm_ad_rgb')
  2. Load the scene with the prb integrator. The tutorial loads the Cornell box from its scenes folder; point the path at wherever your copy of cbox.xml lives.
    scene = mi.load_file('../scenes/cbox.xml', res=128, integrator='prb')
  3. Render the reference image.
    image_ref = mi.render(scene, spp=512)
  4. Perturb the parameter you want to recover. Save the true value first, then overwrite it.
    params = mi.traverse(scene)key = 'red.reflectance.value'param_ref = mi.Color3f(params[key])params[key] = mi.Color3f(0.01, 0.2, 0.9)params.update();
  5. Create the optimizer.
    opt = mi.ad.Adam(lr=0.05)opt[key] = params[key]params.update(opt);
  6. Define the loss and the iteration count.
    def mse(image):    return dr.mean(dr.square(image - image_ref))iteration_count = 50
  7. Run the optimization loop.
    errors = []for it in range(iteration_count):    image = mi.render(scene, params, spp=4)    loss = mse(image)    dr.backward(loss)    opt.step()    opt[key] = dr.clip(opt[key], 0.0, 1.0)    params.update(opt)    err_ref = dr.sum(dr.square(param_ref - params[key]))    print(f"Iteration {it:02d}: parameter error = {err_ref}", end='r')    errors.append(dr.slice(err_ref))print('nOptimization complete.')

Each pass renders the scene with only 4 samples per pixel, so the gradient is noisy, evaluates the loss against the reference, backpropagates with dr.backward, takes one Adam step, clips the colour to the legal range 0 to 1, and writes the new value back into the scene. The tutorial recommends the prb integrator for scenes without moving objects or cameras. The method is described in Vicini, Speierer and Jakob (2021).

How to read the output

Look at two things: a picture and a curve. First, render the scene again after the loop (mi.render(scene, spp=128)) and convert it with mi.util.convert_to_bitmap; if the optimization worked, the red wall should be restored to its original colour. Second, plot the errors list with matplotlib. A curve that falls steadily toward zero indicates convergence, and in the tutorial’s run the last printed line is Iteration 49: parameter error = [0.00147954], a squared distance between the recovered and the true colour of about 0.0015.

The loss itself is a weaker signal. The image-space error between a 4-samples-per-pixel render and the reference can be dominated by Monte Carlo noise rather than by the parameter being optimized, which is why the tutorial tracks the parameter error instead. That check is only possible because the true colour is known in this synthetic scene. In a real inverse problem you have no ground-truth parameter, so judge the result by rendering the recovered scene with many samples per pixel and comparing it with your photographs, and record the sample counts, learning rate and iteration count so the run can be repeated; see computational reproducibility.

Troubleshooting differentiable rendering tools

Mitsuba 3 reports an unsupported variant

Gradients require a variant with the _ad suffix. If mi.set_variant('llvm_ad_rgb') fails on your machine, switch to another _ad variant such as cuda_ad_rgb, or the reverse, depending on whether you have a supported GPU.

The loss is noisy and does not fall smoothly

This is expected when the loop renders at a low sample count. Track a parameter-level error when you know the target, and use a higher sample count for the final render, as the tutorial does (4 in the loop, 128 for the result).

Recovered colours go out of range

The tutorial clips the optimized value with dr.clip(opt[key], 0.0, 1.0) after each step so that colour values stay legal. Keep a similar constraint for any parameter with a physical range.

nvdiffrast fails to build

Install setuptools, wheel and ninja first, then install with --no-build-isolation, exactly as in the README commands above.

PyTorch3D has no GPU support after a source build

The INSTALL file says CUDA support is built when it is available in PyTorch or when FORCE_CUDA=1 is set. Prebuilt wheels are listed only for an older PyTorch release (1.11.0 with Python 3.8 to 3.9), so on current setups use the conda package on Linux or build from source. If your GPU time is scheduled by a cluster, see requesting a GPU in a Slurm batch job.

Free alternatives to Mitsuba 3 for differentiable rendering

The three tools in this guide are alternatives to one another for different jobs. The table lists where each project’s documentation starts and what it highlights; the data table below compares approach and install commands.

Tool Entry point in the documentation Feature the documentation highlights
Mitsuba 3 Inverse rendering tutorials, starting with gradient-based optimization Forward and reverse mode automatic differentiation through Dr.Jit
nvdiffrast README and the documentation at nvlabs.github.io/nvdiffrast; paper arXiv 2011.03277 High-performance rasterization primitives as PyTorch operations
PyTorch3D Why PyTorch3D page and INSTALL.md Efficient 3D operators and heterogeneous batching alongside the rendering API

For work where the rendering must model light transport, Mitsuba 3’s tutorials cover that range of problems. If you already train PyTorch models and want rasterization inside the training loop, nvdiffrast and PyTorch3D are the tools the documentation positions for that. Mitsuba 3 also lists a PyTorch and Mitsuba interoperability tutorial for combining the two. For related Python tooling that runs on the GPU, see CuPy, and for gradient-based attribution in PyTorch see saliency maps.

Sources

Data: Open-source differentiable rendering tools: approach and installation

Compares Mitsuba 3, nvdiffrast and PyTorch3D on how each project's own documentation describes its rendering approach, its underlying framework and its documented install route, compiled from the projects' official docs.

Period covered: 2026.

Open-source differentiable rendering tools: approach and installation
ToolRendering approach per documentationBuilt onDocumented install command
Mitsuba 3Differentiable rendering for inverse problems involving light; gradient integrators such as prb (Path Replay Backpropagation)Dr.Jitpip install mitsuba
nvdiffrastRasterization-based differentiable rendering primitivesPyTorchpip install git+https://github.com/NVlabs/nvdiffrast.git --no-build-isolation
PyTorch3DModular differentiable rendering API within a 3D deep learning libraryPyTorchconda install pytorch3d -c pytorch3d (Linux, CUDA)

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about Differentiable Rendering: Mitsuba 3, nvdiffrast and PyTorch3D

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →