Skip to content

Calibration graph

Use this page to see everything MVIS solves at once, and in what order it gets there.

A factor graph links the unknowns (variables) to the measurements and priors that constrain them (factors). Each factor adds one term to the cost; the solver moves all variables together until the total is smallest. That joint solve is what lets a camera’s time offset, an IMU’s scale and a lever arm be fitted from one recording.

Symbols follow Notation & glossary. The factor equations are on Camera–IMU calibration math, Base IMU factor and Auxiliary IMU factor.

Camera–IMU factor graphA row of IMU states linked by IMU factors. Reprojection factors tie each frame to its anchor state, the camera calibration and the board. The IMU factors share the base IMU intrinsics and t_ga. An auxiliary IMU factor spans three intervals and ties to the auxiliary IMU calibration. A gauge prior holds the first state.camera i: extrinsic, td, tr, f, dboard poseboard shapestate 0state 1state 2state 3state 4base IMU intrinsics, t_gaaux IMU j calibrationreprojectionIMUgaugeaux IMU■ factor · ▢ variable · dashed: only with estimate.board_points

The camera–IMU graph for one camera, one auxiliary IMU and five states. Each state holds the base-IMU pose, velocity and biases. Not drawn, to keep it readable: the reprojection factors also touch the anchor’s velocity and biases, the IMU intrinsics and t_ga; the auxiliary IMU has its own velocity and bias at each of its knots. The IMU calibration variables have priors; camera calibration gets one only on what is held (see Priors below).

VariableSymbolSizeHow many
IMU poseIkGT{}^{G}_{I_k}\mathbf{T}6one per state, at solve.state_hz (30 Hz) on the first camera’s frames
IMU velocity and biasesGvIk{}^{G}\mathbf{v}_{I_k}, bg,k\mathbf{b}_{g,k}, ba,k\mathbf{b}_{a,k}9one per state
Board poseBGT{}^{G}_{B}\mathbf{T}6one; the board stays still
Board shapehmh_m, one per board corner mm (Notation’s hkh_k)1 per corneronly with estimate.board_points: true
Camera calibrationCiIT{}^{I}_{C_i}\mathbf{T}, td,it_{d,i}, tr,it_{r,i}, fi\mathbf{f}_i, di\mathbf{d}_i8 + intrinsicsone per camera
Base IMU intrinsicsDw\mathbf{D}_w, Da\mathbf{D}_a, Tg\mathbf{T}_g (and a triad rotation)24one
Gyro-to-accel offsettgat_{ga}1one per IMU, when estimate_t_ga is on (the default)
Auxiliary IMU calibrationIsjIT{}^{I}_{I_{s_j}}\mathbf{T}, td,jt_{d,j}, its intrinsics31one per auxiliary IMU
Auxiliary IMU velocity and biasesGvIsj{}^{G}\mathbf{v}_{I_{s_j}}, bg,s\mathbf{b}_{g,s}, ba,s\mathbf{b}_{a,s}9 (3 if gyro-only)one per auxiliary knot

x\mathbf{x} stacks all of them. A 60 s recording at 30 Hz has 1800 states, each with a pose, a velocity and two biases.

FactorLinksWeight
Reprojectionone frame’s corners to its anchor state, the board, the camera calibration, the IMU intrinsics and t_ga1 px per axis; Huber past 2 on the frame’s stacked residual
Base IMUtwo consecutive states, the IMU intrinsics, t_gathe integration covariance
Bias random walkthe bias rows of the base and auxiliary IMU factorsthe random-walk densities
Auxiliary IMUtwo base states 3 intervals apart, the auxiliary states and calibrationthe integration covariance
Priorssingle variables (below)per prior

Rates. Each camera adds about 10 reprojection factors per second (solve.factor_hz, settable per camera). The state rate stays 30 Hz, so the trajectory stays dense even when the cameras are thinned.

Priors:

  • Gauge on the first pose: a gauge prior fixes directions the data cannot see. Here it holds the first pose’s position and its rotation about gravity. The tilt (the direction of gravity in the IMU frame) stays free, since the accelerometer sees gravity.
  • First velocity and biases: weak (0.5 m/s, 0.01 rad/s, 0.1 m/s²).
  • IMU intrinsics: a tight prior on the pinned blocks, a weak one on the freed blocks (which blocks).
  • t_ga: zero-mean, 10 ms.
  • Camera calibration: free, except a tight prior on whatever is held: tr,it_{r,i} on a global-shutter camera, and any estimate.camera.*: false.
  • Board shape: each height has a 3 mm prior. A gauge prior holds the mean height and the two tilts at 0, so the board pose keeps the plane and the heights keep only the shape.

The solve minimises the weighted sum of all factor errors, given on Camera–IMU calibration math. After the solve, MVIS computes the marginal covariance of every calibration variable. Its square root is the 1σ (one standard deviation) printed next to each value. A wide 1σ is how a weakly observed value shows itself.

mvis cam-imu reaches the joint solve in steps. Each step gives the next a starting point close enough to converge.

StageWhat happensWhat is estimated
1. WindowBags longer than 80 s: keep the 80 s with the most gyro and accelerometer excitation (input.window_mode: motion). Shorter bags: from 3 s in (input.t0).—
2. DetectionBoard corners in every frame.—
3. Frame selectionKeep frames with enough corners and a valid board pose (PnP: the camera pose from known board points); thin each camera to factor_hz.the board pose per frame, from the camera prior
4. Start valuesPer camera: time offset from angular rates, rotation and gyro bias from a rotation fit, then a linear solve for gravity, scale, velocities and lever arm; the accelerometer bias starts at 0. Per auxiliary IMU: offset, rotation and gyro bias from gyro-to-gyro.rough td,it_{d,i}, CiIR{}^{I}_{C_i}\mathbf{R}, IpCi{}^{I}\mathbf{p}_{C_i}, gyro bias, gravity direction; IsjIR{}^{I}_{I_{s_j}}\mathbf{R}, td,jt_{d,j}
5. Joint solveThe graph above, Levenberg–Marquardt, up to 50 iterations. t_ga re-integration as needed.everything at once
6. Clock retiming (opt-in)For cameras marked retime_sensor_clock: a first solve gives each frame its own capture offset, a regular sensor clock is fitted to them, the frames are re-stamped, and the joint solve runs again.per-frame offsets, then everything
7. ResultsResiduals; χ² per degree of freedom per factor type (the weighted squared error divided by its degrees of freedom, about 1 when the noise model fits); 1σ for every value.—

Checks on the way:

  • Start values: a lever-arm start with 1σ over 5 cm starts at 0 instead. The start reports its gravity magnitude and scale (both should come out near 9.81 m/s² and 1) and how sharp the time-offset minimum is.
  • The joint solve: Levenberg–Marquardt can stop at its damping cap far from a minimum, which looks like convergence. MVIS spots this from the step the linear system still predicts, solves again from the same start with a different damping, and keeps that result only if its cost is lower. A solve whose linear system cannot be solved (no step at all) is marked not ok.

mvis cam has no IMU. Its graph holds one board pose per timestamp, the intrinsics of every camera, the extrinsic of every camera to the first one, the rolling-shutter readout, the board shape when asked, and time offsets between cameras when optimize_time_offset is on (off by default).

StageWhat happensWhat is estimated
1. DetectionBoard corners in every frame.—
2. Start valuesPer camera: OpenCV intrinsics (radtan) or a vanishing-point start (other models), and a board pose per frame.intrinsics, board poses
3. GatingDrop frames with too few tags (6) or too small a spread, poses with a PnP error over 10 px, corners seen in too few frames.—
4. Per-camera solveOne graph per camera.intrinsics, distortion, board poses, readout of rolling-shutter cameras
5. AlignmentAlign the board-pose tracks of the cameras.camera-to-camera extrinsics, time offsets
6. Joint solveAll cameras in one graph.everything at once
7. Outlier rerunDrop corners and frames whose error is over 2 px and above the 99th percentile; solve again from the previous values. On by default.everything again

The camera-only solves after the start values thin the corners to two per tag (corner_subsample: two_anti); the start values use every corner. The camera–IMU solve keeps all of them.

  • Record about 60 s. Past 80 s, MVIS keeps the 80 s with the most motion.
  • Read the 1σ, not only the residual. A small residual with a wide 1σ means the value was not observed: see Degeneracy.
  • Leave solve.state_hz at 30 Hz. Lower rates undersample fast motion (step 4).