Skip to content

Camera model

Use this page to pick the camera model for each camera in your rig, and to see the exact equations MVIS uses.

A camera model Π\Pi maps a 3D point P=[X Y Z]⊤\mathbf{P} = [X\ Y\ Z]^\top in the camera frame {Ci}\{C_i\} to a pixel [u v]⊤[u\ v]^\top: a projection plus a lens distortion. Symbols follow Notation & glossary.

Each model lives in one class under src/mvis/camera/. The class holds the projection, its Jacobians (the derivatives the optimiser uses) and the unprojection. So every model works:

  • in camera-only and camera–IMU calibration;
  • with a calibration board of known corner positions, and in feature mode, where the corners’ 3D positions are not given and are solved as well (example: VI-Rig 4 IMUs + 3 cams, ArUco);
  • mixed across cameras in one rig.
ModelConfig valueOriginDistortion coefficients, in file orderSuits
Pinhole + radial-tangentialradtanBrown–Conradyk1 k2 p1 p2Normal lenses
Kannala–Brandt KB4kb4 (also fisheye, kannala_brandt)Kannala–Brandt, θ\theta from atan2k1 k2 k3 k4Fisheye, including corners past 90°
EquidistantequidistantKannala–Brandt on X/Z,Y/ZX/Z, Y/Zk1 k2 k3 k4Fisheye with every corner under 90°
FOVfovDevernay–FaugerasomegaWide-angle, one parameter
Double Spheredouble_sphereUsenko et al. (Basalt)xi alphaVery wide field of view
EUCMeucmKhomutenko et al.alpha betaVery wide field of view
UCMucmUnified model (Mei), α\alpha formalphaVery wide field of view; Kalibr omni / none
Omni + radtanomni_radtanMei, experimentalxi k1 k2 p1 p2Catadioptric, ultra-wide

Every model shares the four intrinsics fu,fv,cu,cvf_u, f_v, c_u, c_v (fx fy cx cy). The distortion coefficients follow them.

config.yaml
cameras:
- camera_id: 0
distortion_model: kb4

Each model ends with the same affine step from a normalised point [mx my]⊤[m_x\ m_y]^\top to pixels:

u=fu mx+cu,v=fv my+cvu = f_u\, m_x + c_u, \qquad v = f_v\, m_y + c_v

What differs is how [mx my][m_x\ m_y] comes from P\mathbf{P}. A point outside a model’s valid domain has no pixel. That corner is left out of the error statistics and, in camera–IMU, out of the cost. Left uncounted, it would make the error look smaller than it is. So the result counts dropped corners:

  • past_fold: a Kannala–Brandt corner past the fold;
  • other_invalid: anything else, such as Z≤0Z \le 0 under equidistant.

Source: src/mvis/camera/pinhole_radtan.py (GTSAM Cal3DS2).

x=XZ,y=YZ,r2=x2+y2x = \frac{X}{Z},\quad y = \frac{Y}{Z},\quad r^2 = x^2 + y^2 mx=x (1+k1r2+k2r4)+2p1xy+p2 (r2+2x2)my=y (1+k1r2+k2r4)+p1 (r2+2y2)+2p2xy\begin{aligned} m_x &= x\,(1 + k_1 r^2 + k_2 r^4) + 2 p_1 x y + p_2\,(r^2 + 2x^2) \\ m_y &= y\,(1 + k_1 r^2 + k_2 r^4) + p_1\,(r^2 + 2y^2) + 2 p_2 x y \end{aligned}

Valid: Z>0Z > 0.

Both use the Kannala–Brandt odd polynomial in the incidence angle θ\theta (the angle of the ray off the optical axis):

θd=θ (1+k1θ2+k2θ4+k3θ6+k4θ8)\theta_d = \theta\,(1 + k_1\theta^2 + k_2\theta^4 + k_3\theta^6 + k_4\theta^8) mx=θdρ X,my=θdρ Y,ρ=X2+Y2m_x = \frac{\theta_d}{\rho}\, X, \qquad m_y = \frac{\theta_d}{\rho}\, Y, \qquad \rho = \sqrt{X^2 + Y^2}

The two differ only in how they get θ\theta:

Modelθ\thetaValid domain
kb4θ=atan2⁡(ρ,Z)\theta = \operatorname{atan2}(\rho, Z)0≤θ<π0 \le \theta < \pi and θ\theta below the first fold θf\theta_f
equidistantθ=arctan⁡(ρ/Z)\theta = \arctan(\rho / Z), after dividing by ZZZ>0Z > 0, so θ<90°\theta < 90°

The first fold θf\theta_f is the smallest θ>0\theta > 0 where dθd/dθ=1+3k1θ2+5k2θ4+7k3θ6+9k4θ8=0\mathrm{d}\theta_d / \mathrm{d}\theta = 1 + 3k_1\theta^2 + 5k_2\theta^4 + 7k_3\theta^6 + 9k_4\theta^8 = 0 (π\pi when there is none). Past it, θd\theta_d falls again and two rays share one pixel, so the inverse is no longer unique.

  • Below 90° the two models give the same pixels for the same coefficients.
  • At or past 90° only kb4 has a pixel. This is why fisheye means kb4 (Why kb4 for fisheye).
  • After a solve MVIS logs the fold angle and the radius in pixels where the valid domain ends. It warns when that radius lies inside the image.

Source: src/mvis/camera/kb4.py, src/mvis/camera/kb_domain.py, src/mvis/camera/pinhole_equidistant.py (GTSAM Cal3Fisheye).

Source: src/mvis/camera/fov_model.py.

x=XZ,y=YZ,ru=x2+y2,rd=1ωarctan⁡ ⁣(2 rutan⁡ω2)x = \frac{X}{Z},\quad y = \frac{Y}{Z},\quad r_u = \sqrt{x^2 + y^2},\quad r_d = \frac{1}{\omega}\arctan\!\Big(2\, r_u \tan\frac{\omega}{2}\Big) mx=rdru x,my=rdru ym_x = \frac{r_d}{r_u}\, x, \qquad m_y = \frac{r_d}{r_u}\, y

On the optical axis the scale rd/rur_d / r_u is its limit 2tan⁡(ω/2)/ω2\tan(\omega/2)/\omega. As ω→0\omega \to 0 the model becomes a plain pinhole. Valid: Z>0Z > 0.

Source: src/mvis/camera/double_sphere.py.

d1=∥P∥,d2=X2+Y2+(ξd1+Z)2d_1 = \lVert \mathbf{P} \rVert, \qquad d_2 = \sqrt{X^2 + Y^2 + (\xi d_1 + Z)^2} w=α d2+(1−α)(ξd1+Z),mx=Xw,my=Yww = \alpha\, d_2 + (1 - \alpha)(\xi d_1 + Z), \qquad m_x = \frac{X}{w}, \quad m_y = \frac{Y}{w}

Valid: w>0w > 0. Expected ranges: ξ∈[−1,1]\xi \in [-1, 1], α∈[0,1]\alpha \in [0, 1].

Source: src/mvis/camera/eucm.py.

d=β (X2+Y2)+Z2,w=α d+(1−α)Z,mx=Xw,my=Ywd = \sqrt{\beta\,(X^2 + Y^2) + Z^2}, \qquad w = \alpha\, d + (1 - \alpha) Z, \qquad m_x = \frac{X}{w}, \quad m_y = \frac{Y}{w}

Valid: w>0w > 0. Expected ranges: α∈[0,1]\alpha \in [0, 1], β>0\beta > 0.

Source: src/mvis/camera/ucm.py. UCM is EUCM with β=1\beta = 1:

d=∥P∥,w=α d+(1−α)Z,mx=Xw,my=Ywd = \lVert \mathbf{P} \rVert, \qquad w = \alpha\, d + (1 - \alpha) Z, \qquad m_x = \frac{X}{w}, \quad m_y = \frac{Y}{w}

Valid: Z>−wf dZ > -w_f\, d, with wf=(1−α)/αw_f = (1-\alpha)/\alpha for α>0.5\alpha > 0.5, else wf=α/(1−α)w_f = \alpha/(1-\alpha). Past that bound the image folds back. Expected range: α∈[0,1)\alpha \in [0, 1).

Kalibr’s omni camera with distortion none is the same model in ξ\xi form. MVIS converts on import and export only:

ξ=α1−α,γ=f1−α\xi = \frac{\alpha}{1 - \alpha}, \qquad \gamma = \frac{f}{1 - \alpha}

Source: src/mvis/camera/omni_radtan.py. A projection onto the unit sphere with mirror parameter ξ\xi, then radtan distortion on the normalised plane:

d=∥P∥,x=XZ+ξd,y=YZ+ξdd = \lVert \mathbf{P} \rVert, \qquad x = \frac{X}{Z + \xi d}, \quad y = \frac{Y}{Z + \xi d}

then mx,mym_x, m_y from x,yx, y exactly as in radtan. Valid: Z+ξd>0Z + \xi d > 0. Expected range: ξ∈[0,5]\xi \in [0, 5]. This is Kalibr’s omni / radtan.

All projection kernels outside GTSAM are in src/mvis/calibration/factors/custom_projections.py, with analytical Jacobians.

More parameters are not better. A model richer than the lens needs fits noise. On wide field-of-view data it can slide into a basin (a wrong local minimum) that looks converged and is not.

Choose by the largest incidence angle of the board corners, not by the lens’s advertised field of view. Every result reports this angle (max_theta_deg, and max_theta_used_deg for the corners the fit kept), and counts any corners past the model’s valid domain.

  • Normal lens: radtan.
  • Fisheye: kb4, which is what fisheye and kannala_brandt select. It stays valid past 90° up to its first fold (why).
  • Every corner under 90°, and you want equidistant by name: equidistant. It gives the same pixels as kb4 there.
  • Very wide field of view, a different shape: eucm, ucm or double_sphere.
  • Catadioptric (lens plus mirror): omni_radtan. It is experimental.

The camchain written for Kalibr uses these names:

MVISKalibr camera_model / distortion_model
radtanpinhole / radtan
equidistantpinhole / equidistant
kb4pinhole / equidistant: exact for Z>0Z > 0; MVIS warns when the fit used corners at or past 90°, which Kalibr cannot project
fovpinhole / fov
double_sphereds / none
eucmeucm / none
ucmomni / none, converted to ξ\xi form
omni_radtanomni / radtan

Camera-only calibration runs two passes: a first solve, then a second one after outlier rejection. A guardrail keeps the first-pass result if the second pass lands on any of these (src/mvis/calibration/pipeline_cam.py):

  • a focal length of 1 px or less;
  • fuf_u and fvf_v more than 30 % apart;
  • a distortion coefficient above 2 in magnitude (for ucm: α\alpha outside [0,1)[0, 1); for omni_radtan: ξ\xi outside [0,5][0, 5]);
  • a 95th-percentile reprojection error more than 1.5 times the first pass’s, on a camera whose fuf_u moved less than 5 %;
  • a mean RMS reprojection error over all cameras more than 1.5 times the first pass’s.

It is a backstop, not a substitute for picking the right model.

A rolling-shutter camera exposes its rows one after another, so one image spans a range of times. The readout trt_r is the time from the first row to the last.

A corner observed in row vv of an image of height HH was captured at

tI=tC+td+η(v) trt_I = t_C + t_d + \eta(v)\, t_r

The row factor η(v)\eta(v) depends on which row the driver stamps (rolling_mode, default 0):

rolling_modeFrame stamp atη(v)\eta(v)
0top rowv/Hv / H
1middle rowv/H−0.5v / H - 0.5
2bottom row(v−H)/H(v - H) / H

Source: src/mvis/core/shutter.py. For a global-shutter camera tr=0t_r = 0, so every corner of a frame shares one time.

The camera–IMU vision factor uses that per-corner time directly. It takes the pose of {I}\{I\} at tIt_I from the base IMU, composes the extrinsics, and then projects:

zi,k=Π(Cipk(tI))+n\mathbf{z}_{i,k} = \Pi\big({}^{C_i}\mathbf{p}_{k}(t_I)\big) + \mathbf{n}

where Cipk(tI){}^{C_i}\mathbf{p}_k(t_I) is board corner kk in the camera frame at the corner’s capture time. Projection, time offset, readout and extrinsics sit in one factor, so tdt_d and trt_r get Jacobians and are solved with everything else, not corrected afterwards. The HTML report uses the same projection code, so the residuals it shows are the ones the optimiser minimised.

Turn it on per camera:

config.yaml
cameras:
- camera_id: 0
shutter_type: rolling

Or through the estimate: block, with estimate.camera.tr: true. Practical guidance: Rolling shutter.