Skip to content

Evaluation & metrics

Use this page to tell whether a calibration is good. Check it in this order:

  1. Read the report: did the motion exercise every parameter?
  2. Run mvis verdict: are the fit and the values plausible?
  3. Run an independent check: round trip, repeatability, another tool or a digital twin.

The same metrics back the leaderboard, so a local run and a published entry measure the same thing.

Every run writes a self-contained HTML report (what is in it). Read its data quality section before the error numbers. It shows:

  • Motion excitation: grades for how much the motion moved each axis
  • Rotation-axis spread: whether you rotated about all three axes
  • Angular-velocity timeline: where in the recording the motion was
  • Image coverage: where the board corners fell in each image
Terminal window
mvis verdict results/run/t265_cam_calib.yaml # camera-only
mvis verdict results/run/cam_imu.json -c cam_imu.yaml # camera–IMU, with its reference: block

Each check is pass, warn or fail. The grade is RED if any check fails, YELLOW if any warns, GREEN otherwise. The camera–IMU HTML report uses the same thresholds for its badges.

CheckidPassWarnFail
Reprojection medianreproj.median≤ 1.41 px≤ 2.83 pxabove
Reprojection P95reproj.p95≤ 2.83 px≤ 5.66 pxabove
Reprojection RMS (when no median)reproj.rms≤ 1.41 px≤ 2.83 pxabove
Camera and IMU fit, χ²/doffit.cam_chi2_dof, fit.imu_chi2_dof, fit.aux_imu_chi2_dof≤ 1.5≤ 3.0above
Focal lengthintr.focal_positivefx, fy > 0—otherwise
Focal aspectintr.focal_aspect|fx/fy − 1| ≤ 0.02above—
Principal pointintr.principal_pointwithin 10 % of the image size from the centrefurther—
Distortionintr.distortion_finitefinite—not finite
Reprojection presentreproj.missingevery camera has errors—a camera has none
Cameras in the resultresult.camerasat least one—none
Corner observations per camera (camera-only)data.observations≥ 500fewer—
IMU scale (camera–IMU)imu.dw_scale, imu.da_scale|diagonal − 1| ≤ 0.05above—
Camera–IMU time offsettime.td|td| ≤ 50 msabove—
Solver statussolve.oktrue—false
Iterationssolve.convergedbelow max_iterationshit the cap—
Solver stepsolve.lm_steptook at least one steptook no steptook no step and failed
Lever against reference:ref.lever≤ 10 mmabove—
Baseline against reference:ref.baseline≤ 5 mmabove—

Notes:

  • Pixel units: the reprojection thresholds are on the length of each corner’s 2D error, as Kalibr reports it. They are per-axis limits times √2: 1 and 2 px for the median and RMS, 2 and 4 px for the P95.
  • χ²/dof: the solve’s cost per degree of freedom. Near 1 means the noise settings match the data.
  • Warnings can be real: a raw consumer IMU can have a scale error above 5 %, and a USB camera a td above 50 ms. Confirm them across recordings.
  • ok: true from the solver only means it finished and passed its own gates. verdict never counts it as proof of a good fit.

mvis verdict --independent-check <kind> records which of these you ran.

Terminal window
mvis roundtrip -c cam_imu.yaml

It needs no ground truth. MVIS applies the calibration to its own bag and calibrates the corrected bag again. PASS needs every one of these on the corrected bag:

QuantityExpected after correctionTolerance
Dw, Da diagonals1within 0.01
Gyroscope and accelerometer bias0below 0.02 (rad/s, m/s²)
Lever and stereo baselineunchangedwithin 2 mm
td0above 2 ms is a warning only, because td can be weakly observable

The IMU terms must collapse to identity and zero while the geometry stays the same. See Round-trip validation.

Calibrate several recordings of one rig with input.bags and compare the results. A parameter that moves between recordings was not pinned down by the motion in them. See Camera–IMU calibration.

Run Kalibr or Basalt on the same recordings. mvis.io.kalibr reads Kalibr’s camchains and results (Kalibr files). The leaderboard shows MVIS, Kalibr and Basalt side by side.

A digital twin replays a real recording’s motion, frame times and corner layout with known parameters. Solving it measures bias against a truth no one gave the solver. See Digital-twin check.

The leaderboard and each dataset’s eval page use these numbers. Methodology explains how to read them.

MetricUnitWhat it is
Median ± sd (n)per parameterMedian and standard deviation of one tool’s estimate over the n recordings it solved
Differenceper parameterDifference of two tools’ medians; ◆ when it is larger than the larger of the two spreads
Focal length, principal pointpxCamera intrinsics
Lever armmmCamera position in the base IMU frame, or auxiliary IMU position
Rotationradlog(R) rotation vector of an extrinsic
Stereo baselinemmDistance between two cameras
td, t_gamsCamera–IMU and gyro-to-accelerometer time offsets
Solvedn of NRecordings a tool solved

The benchmark datasets have no ground truth. A small spread shows repeatability, not accuracy, and a difference shows that two tools disagree, not which one is right. Reprojection error is not compared across tools, because it measures fit.

Where a recording has published or factory values, the regression suite reports the percentage difference to them (--gt). These values are compared, never used as a pass or fail gate. They are kept with the regression suite.

We run MVIS over real recordings and digital twins, and gate every change against stored baselines with per-metric tolerances (focal length, distortion, extrinsics, time offsets, readout, reprojection). The runner and its baselines stay with the maintainers, not in the public repository; the commands below are how we run it. See Regression testing for the tests you can run.

Terminal window
python scripts/run_regression.py --mode all --level fast # the gate
python scripts/run_regression.py --only-collection mvis_4imus_3cams/april_01
python scripts/run_regression.py --mode all --level all --dry # compare existing results only
--levelRuns
fastThe first recording of each family: the gate for every change
midThree recordings per family
fullEvery recording of every family
optionalRows outside the gates, such as camera-model variants
allfull and optional

--help prints the current row counts. --update-baseline rewrites the stored baselines after an intended change; the baseline diff then shows every number the change moved. How the suite is built: Regression testing.