Skip to content

Digital-twin check

Step 7 of the walkthrough. Here you test the solver on a simulated copy of your recording, where the truth is known.

  • You need: the camera–IMU config and bag from step 4, and MVIS installed from source or from the package.
  • You get: the error of each calibration parameter against the truth.
  • Check: the noise-free errors are near zero (section 2).

A digital twin is a simulated copy of your recording:

  • The truth is your real solve: calibration, board pose and shape, biases and trajectory. A small seeded perturbation then moves the camera intrinsics, the camera and aux-IMU extrinsics, each td and each t_ga, by up to 1 % focal, 4 px principal point, 5 % of each distortion coefficient, 10 mrad per rotation axis, 5 mm per lever axis, 1 ms td and 0.5 ms t_ga. Readout, IMU intrinsics, biases, gravity, board and trajectory stay as solved. The perturbation makes the solve recover values it was not given, and keeps your real calibration out of a twin you save or share. perturb_seed=None in capture keeps the real solve as it is.
  • The timing is real: MVIS keeps the real frame times, the corner ids seen in each frame and the IMU stamps.
  • The data is new: MVIS regenerates the corners and the IMU readings from the truth, with rolling shutter and t_ga (the time offset between the gyroscope and the accelerometer).
  • The solve is the real pipeline, run again on the new data.

So any gap between the result and the truth comes from the solver, not from the recording.

The twin is the Python module mvis.simulation.twin.

twin.py
from mvis.simulation import twin
# Runs the real solve once (it needs the bag) and takes it as the truth.
# noise_from_residuals sets the noise to match the real residuals.
tw = twin.capture("virig_4imus_4cams_cam_imu.yaml", "virig4",
noise_from_residuals=True)
report = twin.solve(tw, seed=0) # with noise
clean = twin.solve(tw, noise=False) # without noise
tw.save("virig4_twin.npz") # keep it; it holds no images
tw = twin.Twin.load("virig4_twin.npz")
  • capture runs your camera–IMU config once on the real bag. Its outputs go to results/twin_capture (set another folder with work_dir=).
  • noise_from_residuals=True measures the corner noise from the real solve. It runs one extra solve of the twin to scale the IMU noise, so the twin’s IMU fit matches the real one.
  • solve simulates, solves and returns the errors (estimate minus truth).
  • seed picks the noise draw. Run a few seeds, not one.
  • A saved twin needs no bag. Solve it again after any change to MVIS or to the solve settings.

Each solve returns, per sensor:

  • Cameras: lever arm (mm), rotation as a log(R) vector (mrad), td (the camera–IMU time offset, ms), readout (ms), focal length (px).
  • Base IMU: t_ga (ms), gyroscope and accelerometer scale (%).
  • Each aux IMU: lever, rotation, td, t_ga and scales.
  • Board: rms height error (mm), when the board shape is estimated.

Read the two kinds of run side by side:

  • The noise-free error should be near zero. If it is not, the solver does not recover the truth even from perfect data. That is a model or observability problem.
  • The error with noise shows what your data can support. Large errors with noise and small noise-free errors mean the data limits the result, not the solver.

We ran the step-4 config (all 4 cameras, all 4 IMUs, bag 01) and simulated the same 80 s from its result. Each video shows:

  • Left: the IMU trajectory and the board, with each camera’s pose.
  • Right: each camera’s corners, observed (circles) and reprojected (dots), with the RMS for that frame.
  • Bottom: position (m) and orientation as log(R) from the start (rad), with a cursor at the current time.
The real solve. Corners as the cameras saw them. Reprojection RMS over the run: 0.52 px.
The digital twin. Every corner simulated from the solved calibration and trajectory, at 30 Hz, with 0.5 px noise.

This video twin is simpler than the twin module of section 1:

  • Truth: the real solve as it is, with no perturbation: trajectory, board, every camera and IMU.
  • Cameras: all four at 30 Hz, each seeing every tag in its view, with 0.5 px corner noise.
  • IMUs: 200 Hz, with a fixed noise model (gyroscope 1.7e-4 rad/s/√Hz, accelerometer 3e-3 m/s²/√Hz).

So its corners are cleaner than the real ones. On real data the per-frame RMS jumps when the board is blurred or far away.

Solved against its truth (one noise draw, all 4 cameras and all 4 IMUs, reprojection RMS 0.71 px):

SensorLever errorRotation errorReadout error
T265 left0.13 mm0.17 mrad—
T265 right0.14 mm0.14 mrad—
ELP-L0.16 mm0.12 mrad0.05 ms
ELP-R0.14 mm0.16 mrad0.04 ms
GX3-25 (to the base IMU)0.18 mm0.54 mrad—
GX3-35 (to the base IMU)0.24 mm0.23 mrad—
T265 IMU (to the base IMU)0.18 mm0.45 mrad—

Every lever lands within 0.25 mm and every rotation within 0.6 mrad of the truth. The solve converges on the full rig.

We built a twin module twin of the ELP-pair camera–IMU run on virig-4imus-4cams bag 01:

  • Cameras: ELP left and right, rolling shutter.
  • IMUs: the T265 IMU as base, the Xsens as aux IMU.
  • Settings: as in the “ELP RS stereo + two IMUs” run on the results page: camera intrinsics free, t_ga on, board shape on.

Errors against the truth (estimate minus truth). For lever and rotation the table gives the length of the 3D error; for a scale, the largest of its three axis errors.

ParameterNoise-freeWith noise, RMS over seeds 0–2
Camera lever0.094 mm0.44 mm
Camera rotation0.177 mrad0.32 mrad
Camera td0.007 ms0.024 ms
Camera readout0.003 ms0.017 ms
Camera focal length0.009 px0.096 px
Base t_ga0.015 ms0.053 ms
Base gyroscope scale0.011 %0.108 %
Base accelerometer scale0.013 %0.033 %
Board height rms0.23 mm0.23 mm
Aux lever0.164 mm0.20 mm
Aux rotation0.245 mrad0.58 mrad
Aux td0.007 ms0.036 ms

The camera rows are ELP left; ELP right is close. What it shows:

  • The solver converges to the truth. Without noise every error is a fraction of a mm, mrad or ms, and the scales land within 0.02 %. This is the check that matters.
  • Noise sets the limit, and it matches real data. With noise, the gyroscope scale lands within 0.11 %, t_ga within 0.05 ms and the ELP readout within 0.02 ms. That is no larger than the real spread over the 15 recordings, which also vary the motion: T265 gyroscope scale sd 0.03–0.13 % per axis, t_ga sd 0.13 ms, ELP readout sd 0.12 ms (results page, “ELP RS stereo + two IMUs”).
  • The board height keeps about 0.2 mm of error even without noise.

A check for every MVIS change. We keep these errors as a reference and re-run the twin for new MVIS versions. A version fails if a solve fails, takes no step, or an error grows past the reference by more than:

QuantityAllowed growth (noise-free)
Lever0.2 mm
Rotation0.2 mrad
td, readout0.05 ms
t_ga0.1 ms
Gyroscope scale0.01 %
Accelerometer scale0.02 %
Focal length0.2 px
Board height rms0.1 mm

Errors with noise may grow by twice these amounts, as three seeds are a rough sample. Smaller errors never fail.

We also used twins to choose the recording motion of step 2. A simulated Looper Insight 9 ran each motion on 3 noise seeds and once without noise. See how we judged it.

On real data there is no truth. Two checks stand in for it, strongest first:

  1. Round trip: mvis roundtrip applies the calibration to its own bag and calibrates again. PASS needs the IMU intrinsics to collapse and the geometry to hold.
  2. Repeatability: calibrate several recordings of the same rig with batch mode. A parameter that moves between recordings was not pinned down by their motion.

To try a rig or a motion you have not recorded, start from a simulator example. Each one holds a real motion and a full rig setup (cameras, IMUs, board), with a perturbed truth like a twin:

sim.py
from mvis.simulation.examples import load_example, simulate
from mvis.calibration.pipeline_cam_imu import run_cam_imu_calibration_multi
ex = load_example("twin_gs") # ships with the package
data = simulate(ex, seed=0) # cams, t_imu, w_m, a_m, aux_imus
result = run_cam_imu_calibration_multi(
data["cams"], ex.board_points_flat, data["t_imu"], data["w_m"], data["a_m"],
aux_imus=data["aux_imus"] or None)
  • In the package: twin_gs, a global-shutter stereo twin with one IMU.
  • In a source checkout (data/sim_examples/): more examples, among them virig_4imus_3cams (our VI-Rig), elp_t265, tum_vi_raw, twin_rs (a rolling-shutter twin), guide06c_60 (the AprilGrid guide motion of step 2) and guide60b (an earlier chessboard motion). mvis.simulation.examples.available() lists what your install has.
  • Licences: each example’s setup.yaml names its licence; check it before you reuse a guide motion.
  • Lower level: mvis.simulation.scenario builds a dataset from a trajectory, a true calibration and a board placed in the world.

For the theory of weak motions, see Degeneracy.

Step 8: round-trip validation checks the calibration on real data.