Skip to content

Camera–IMU calibration math

Use this page to see what the camera–IMU solve minimises and how a board corner ties the cameras to the IMU trajectory. The two IMU factors have their own pages:

  • Base IMU factor: links consecutive states of the base IMU. Its intrinsics, biases and t_ga sit inside it.
  • Auxiliary IMU factor: each extra IMU sees the same motion through its own extrinsic, time offset and intrinsics.

A factor is one term of the cost. A residual is the error one factor measures. The base IMU is the first IMU in the config; its clock is the reference clock, and the trajectory is its pose over time. Symbols follow Notation & glossary.

The solve finds the one set of values that makes every measurement fit best, given the priors. It minimises:

min⁡x  ∑i,fρ ⁣(∥rcami,f∥σpx2)+∑k∥rimuk∥Pk2+∑j,k∥rauxj,k∥Pj,k2+∑p∥rpriorp∥Σp2\min_{\mathbf{x}}\; \sum_{i,f}\rho\!\left(\big\|\mathbf{r}_{\text{cam}}^{i,f}\big\|^2_{\sigma_{px}}\right) +\sum_{k}\big\|\mathbf{r}_{\text{imu}}^{k}\big\|^2_{\mathbf{P}_k} +\sum_{j,k}\big\|\mathbf{r}_{\text{aux}}^{j,k}\big\|^2_{\mathbf{P}_{j,k}} +\sum_{p}\big\|\mathbf{r}_{\text{prior}}^{p}\big\|^2_{\boldsymbol{\Sigma}_p}
  • x\mathbf{x}: everything solved: the IMU states, the board pose, and every calibration value. The full list is on Calibration graph.
  • i,fi, f: camera, frame. mm: a board corner. Notation writes the corner index as kk; here kk is taken by the IMU states, so corners use mm (hmh_m and zi,m\mathbf{z}_{i,m} are Notation’s hkh_k and zi,k\mathbf{z}_{i,k}).
  • rcami,f\mathbf{r}_{\text{cam}}^{i,f}: the residuals of all corners mm of frame ff in camera ii, stacked into one vector (2 rows per corner).
  • kk: an interval between two IMU states. jj: an auxiliary IMU.
  • ρ\rho: the Huber loss. It acts on a frame’s stacked residual, not on single corners: past a whitened norm of 2 (2 px at σpx\sigma_{px} = 1 px) the error grows linearly instead of quadratically, so a frame with a large total error is down-weighted as a whole (solve.huber_px, default 2.0).
  • ∥r∥Σ2=r⊤Σ−1r\|\mathbf{r}\|^2_{\boldsymbol\Sigma} = \mathbf{r}^\top\boldsymbol\Sigma^{-1}\mathbf{r}: the squared error weighted by its covariance. Pixels use σpx\sigma_{px} = 1 px (solve.sigma_px). The IMU terms use P\mathbf{P}, the covariance the integration propagates from the IMU noise densities.

The solver is Levenberg–Marquardt (a damped Gauss–Newton method) with analytic Jacobians (the derivatives of each residual with respect to each variable).

The reprojection factor asks: given the current estimate, where should board corner mm appear in the image, and how far is that from where it was detected?

A frame carries one stamp tCt_C on the camera clock. Two things move the real capture time of each corner:

  • Time offset td,it_{d,i}: the camera clock runs offset from the base-IMU clock.
  • Rolling shutter: a rolling-shutter camera exposes its rows one after another. The whole frame takes the readout time tr,it_{r,i}. A corner on row vmv_m of an image HH rows high is captured later than the first row.
tm=tC+td,i+η(vm) tr,i,η(vm)=vmHt_m = t_C + t_{d,i} + \eta(v_m)\, t_{r,i}, \qquad \eta(v_m) = \frac{v_m}{H}

η(v)\eta(v) is the row factor of Notation. η(vm)=vm/H\eta(v_m) = v_m/H is for a stamp at the top row, the default. With solve.rolling_mode: 1 (stamp at the middle row) η(vm)=vm/H−0.5\eta(v_m) = v_m/H - 0.5; with solve.rolling_mode: 2 (stamp at the bottom row) η(vm)=vm/H−1\eta(v_m) = v_m/H - 1. The readout tr,it_{r,i} is solved only for a rolling-shutter camera with estimate.camera.tr: true (off by default). Otherwise the camera is treated as global shutter and tr,it_{r,i} is held at 0.

The IMU states sit at discrete times tkt_k (30 Hz by default, solve.state_hz). A corner’s capture time tmt_m almost never lands on one. MVIS gets the pose at tmt_m from the IMU itself. This is solve.vision_pose: imu, the default.

Take the state kk at or just before the frame (the anchor). Integrate the raw base-IMU samples from tkt_k over the signed interval Δm=tm−tk\Delta_m = t_m - t_k:

IGR(tm)=IkGR  ΔRm{}^{G}_{I}\mathbf{R}(t_m) = {}^{G}_{I_k}\mathbf{R}\;\Delta\mathbf{R}_m GpI(tm)=GpIk+GvIk Δm+12 Gg Δm2+IkGR  Δpm{}^{G}\mathbf{p}_{I}(t_m) = {}^{G}\mathbf{p}_{I_k} + {}^{G}\mathbf{v}_{I_k}\,\Delta_m + \tfrac12\,{}^{G}\mathbf{g}\,\Delta_m^2 + {}^{G}_{I_k}\mathbf{R}\;\Delta\mathbf{p}_m
  • ΔRm\Delta\mathbf{R}_m, Δpm\Delta\mathbf{p}_m: the rotation and position change integrated from the IMU readings over Δm\Delta_m, in the anchor’s frame. They use the corrected readings: current biases, Dw\mathbf{D}_w, Da\mathbf{D}_a, Tg\mathbf{T}_g and t_ga (see Base IMU factor).
  • Gg=[0, 0, −9.81]⊤{}^{G}\mathbf{g} = [0,\ 0,\ -9.81]^\top m/s²: gravity in the global frame, which is gravity-aligned.
  • A negative Δm\Delta_m is integrated backwards in time. The window follows the current td,it_{d,i} and tr,it_{r,i}, so it moves as they change.

The corner pose therefore follows the measured motion between states (for example 200 Hz IMU samples between 30 Hz states). There is no interpolation error for the calibration values to absorb. The IMU velocity and intrinsics enter the reprojection factor too, so the gyro scale is fitted by the camera and the IMU terms together.

To save time the factor integrates once and corrects the result to first order as the estimate moves (solve.vision_imu_linearized, default on). It integrates again when a capture time has moved more than 1 ms.

With the IMU pose at tmt_m, chain the transforms from the board to the camera:

Cipm=(IGT(tm)  CiIT)−1  BGT  (Bpm+hm ez){}^{C_i}\mathbf{p}_m = \big({}^{G}_{I}\mathbf{T}(t_m)\;{}^{I}_{C_i}\mathbf{T}\big)^{-1}\; {}^{G}_{B}\mathbf{T}\;\big({}^{B}\mathbf{p}_m + h_m\,\mathbf{e}_z\big) rcami,f,m=Π(Cipm; fi, di)−zi,m\mathbf{r}_{\text{cam}}^{i,f,m} = \Pi\big({}^{C_i}\mathbf{p}_m;\ \mathbf{f}_i,\ \mathbf{d}_i\big) - \mathbf{z}_{i,m}
  • Bpm{}^{B}\mathbf{p}_m: the corner’s position on the board, from the board config.
  • hmh_m: the corner’s height off the flat board, along the board normal ez\mathbf{e}_z. It is 0 unless estimate.board_points: true (Why calibrate the board shape).
  • BGT{}^{G}_{B}\mathbf{T}: the board pose. The board stays still, so it is one pose for the whole recording.
  • Π\Pi: the camera model, with intrinsics fi\mathbf{f}_i and distortion di\mathbf{d}_i (Camera model).
  • zi,m\mathbf{z}_{i,m}: the detected pixel.

One factor holds all corners of one frame. It touches the anchor pose, the anchor velocity and biases, the board pose, the camera’s calibration (CiIT{}^{I}_{C_i}\mathbf{T}, td,it_{d,i}, tr,it_{r,i}, fi\mathbf{f}_i, di\mathbf{d}_i), the IMU intrinsics, t_ga, and the board heights when they are solved.

The time offsets have a derivative because the pose depends on the capture time:

∂ IGT(tm)∂td,i  ∝  [Iω(tm)GvI(tm)],∂ rcam∂tr,i=η(vm) ∂ rcam∂td,i\frac{\partial\,{}^{G}_{I}\mathbf{T}(t_m)}{\partial t_{d,i}} \;\propto\; \begin{bmatrix} {}^{I}\boldsymbol{\omega}(t_m) \\ {}^{G}\mathbf{v}_{I}(t_m) \end{bmatrix}, \qquad \frac{\partial\,\mathbf{r}_{\text{cam}}}{\partial t_{r,i}} = \eta(v_m)\,\frac{\partial\,\mathbf{r}_{\text{cam}}}{\partial t_{d,i}}

So td,it_{d,i} is seen only while the rig turns or moves, and tr,it_{r,i} only when corners on different rows move differently. Observability works this through.

One could integrate the IMU with fixed values and correct the result later. The solver could not move those values then: the correction has no derivative with respect to them.

MVIS puts every calibration value inside the residual instead:

  • the IMU intrinsics Dw\mathbf{D}_w, Da\mathbf{D}_a, Tg\mathbf{T}_g and the biases;
  • the extrinsics CiIT{}^{I}_{C_i}\mathbf{T} and IsjIT{}^{I}_{I_{s_j}}\mathbf{T};
  • the time offsets td,it_{d,i}, td,jt_{d,j}, t_ga and the readout tr,it_{r,i}.

Each then has a Jacobian, and the optimiser can move it. The price: the IMU integrals depend on the values being solved. The factors carry first-order corrections for small changes and integrate again when a value moves far.

FactorConstrains directlyConstrains through the others
Reprojectioncamera intrinsics, CiIT{}^{I}_{C_i}\mathbf{T}, td,it_{d,i}, tr,it_{r,i}, board pose and shapethe trajectory, and the gyro scale through the corner poses
Base IMUbiases, Dw\mathbf{D}_w, Da\mathbf{D}_a, Tg\mathbf{T}_g, t_gametric trajectory, gravity direction
Auxiliary IMUIsjIT{}^{I}_{I_{s_j}}\mathbf{T}, td,jt_{d,j}, its own intrinsics and t_gathe base trajectory, through the shared motion

No single factor sees the whole state. Which values the data can fix depends on the motion: see Observability and Degeneracy.

What this means for your recording and setup

Section titled “What this means for your recording and setup”
  • Turn and move while the board is in view. The time offsets and the readout get their signal only from motion.
  • Fill the image rows. The readout is fitted from how the error changes from the top row to the bottom row.
  • Give the solve a good camera prior. The camera intrinsics are refined here, but they start from the camera-only result (step 3).
  • Use the cleanest camera as the first camera. Its frames set the state times. A global-shutter, low-latency camera is best (step 4).