Camera–IMU calibration math
Use this page to see what the camera–IMU solve minimises and how a board corner ties the cameras to the IMU trajectory. The two IMU factors have their own pages:
- Base IMU factor: links consecutive
states of the base IMU. Its intrinsics, biases and
t_gasit inside it. - Auxiliary IMU factor: each extra IMU sees the same motion through its own extrinsic, time offset and intrinsics.
A factor is one term of the cost. A residual is the error one factor measures. The base IMU is the first IMU in the config; its clock is the reference clock, and the trajectory is its pose over time. Symbols follow Notation & glossary.
The cost
Section titled “The cost”The solve finds the one set of values that makes every measurement fit best, given the priors. It minimises:
- : everything solved: the IMU states, the board pose, and every calibration value. The full list is on Calibration graph.
- : camera, frame. : a board corner. Notation writes the corner index as ; here is taken by the IMU states, so corners use ( and are Notation’s and ).
- : the residuals of all corners of frame in camera , stacked into one vector (2 rows per corner).
- : an interval between two IMU states. : an auxiliary IMU.
- : the Huber loss. It acts on a frame’s stacked residual, not on single
corners: past a whitened norm of 2 (2 px at = 1 px) the error grows
linearly instead of quadratically, so a frame with a large total error is
down-weighted as a whole (
solve.huber_px, default 2.0). - :
the squared error weighted by its covariance. Pixels use = 1 px
(
solve.sigma_px). The IMU terms use , the covariance the integration propagates from the IMU noise densities.
The solver is Levenberg–Marquardt (a damped Gauss–Newton method) with analytic Jacobians (the derivatives of each residual with respect to each variable).
The reprojection factor
Section titled “The reprojection factor”The reprojection factor asks: given the current estimate, where should board corner appear in the image, and how far is that from where it was detected?
When was the corner captured?
Section titled “When was the corner captured?”A frame carries one stamp on the camera clock. Two things move the real capture time of each corner:
- Time offset : the camera clock runs offset from the base-IMU clock.
- Rolling shutter: a rolling-shutter camera exposes its rows one after another. The whole frame takes the readout time . A corner on row of an image rows high is captured later than the first row.
is the row factor of Notation.
is for a stamp at the top row, the default. With solve.rolling_mode: 1 (stamp at
the middle row) ; with solve.rolling_mode: 2 (stamp at
the bottom row) . The readout is solved only for a
rolling-shutter camera with estimate.camera.tr: true (off by default). Otherwise
the camera is treated as global shutter and is held at 0.
Where was the IMU at that time?
Section titled “Where was the IMU at that time?”The IMU states sit at discrete times (30 Hz by default, solve.state_hz).
A corner’s capture time almost never lands on one. MVIS gets the pose at
from the IMU itself. This is solve.vision_pose: imu, the default.
Take the state at or just before the frame (the anchor). Integrate the raw base-IMU samples from over the signed interval :
- , : the rotation and position change
integrated from the IMU readings over , in the anchor’s frame. They
use the corrected readings: current biases, , ,
and
t_ga(see Base IMU factor). - m/s²: gravity in the global frame, which is gravity-aligned.
- A negative is integrated backwards in time. The window follows the current and , so it moves as they change.
The corner pose therefore follows the measured motion between states (for example 200 Hz IMU samples between 30 Hz states). There is no interpolation error for the calibration values to absorb. The IMU velocity and intrinsics enter the reprojection factor too, so the gyro scale is fitted by the camera and the IMU terms together.
To save time the factor integrates once and corrects the result to first order
as the estimate moves (solve.vision_imu_linearized, default on). It integrates
again when a capture time has moved more than 1 ms.
Project and compare
Section titled “Project and compare”With the IMU pose at , chain the transforms from the board to the camera:
- : the corner’s position on the board, from the board config.
- : the corner’s height off the flat board, along the board normal
. It is 0 unless
estimate.board_points: true(Why calibrate the board shape). - : the board pose. The board stays still, so it is one pose for the whole recording.
- : the camera model, with intrinsics and distortion (Camera model).
- : the detected pixel.
One factor holds all corners of one frame. It touches the anchor pose, the
anchor velocity and biases, the board pose, the camera’s calibration
(, , , , ),
the IMU intrinsics, t_ga, and the board heights when they are solved.
How the time offsets get their gradient
Section titled “How the time offsets get their gradient”The time offsets have a derivative because the pose depends on the capture time:
So is seen only while the rig turns or moves, and only when corners on different rows move differently. Observability works this through.
Why calibration sits inside the factors
Section titled “Why calibration sits inside the factors”One could integrate the IMU with fixed values and correct the result later. The solver could not move those values then: the correction has no derivative with respect to them.
MVIS puts every calibration value inside the residual instead:
- the IMU intrinsics , , and the biases;
- the extrinsics and ;
- the time offsets , ,
t_gaand the readout .
Each then has a Jacobian, and the optimiser can move it. The price: the IMU integrals depend on the values being solved. The factors carry first-order corrections for small changes and integrate again when a value moves far.
What each factor constrains
Section titled “What each factor constrains”| Factor | Constrains directly | Constrains through the others |
|---|---|---|
| Reprojection | camera intrinsics, , , , board pose and shape | the trajectory, and the gyro scale through the corner poses |
| Base IMU | biases, , , , t_ga | metric trajectory, gravity direction |
| Auxiliary IMU | , , its own intrinsics and t_ga | the base trajectory, through the shared motion |
No single factor sees the whole state. Which values the data can fix depends on the motion: see Observability and Degeneracy.
What this means for your recording and setup
Section titled “What this means for your recording and setup”- Turn and move while the board is in view. The time offsets and the readout get their signal only from motion.
- Fill the image rows. The readout is fitted from how the error changes from the top row to the bottom row.
- Give the solve a good camera prior. The camera intrinsics are refined here, but they start from the camera-only result (step 3).
- Use the cleanest camera as the first camera. Its frames set the state times. A global-shutter, low-latency camera is best (step 4).