Evaluation & metrics
Use this page to tell whether a calibration is good. Check it in this order:
- Read the report: did the motion exercise every parameter?
- Run
mvis verdict: are the fit and the values plausible? - Run an independent check: round trip, repeatability, another tool or a digital twin.
The same metrics back the leaderboard, so a local run and a published entry measure the same thing.
1. Read the report first
Section titled “1. Read the report first”Every run writes a self-contained HTML report (what is in it). Read its data quality section before the error numbers. It shows:
- Motion excitation: grades for how much the motion moved each axis
- Rotation-axis spread: whether you rotated about all three axes
- Angular-velocity timeline: where in the recording the motion was
- Image coverage: where the board corners fell in each image
2. Grade it with mvis verdict
Section titled “2. Grade it with mvis verdict”mvis verdict results/run/t265_cam_calib.yaml # camera-onlymvis verdict results/run/cam_imu.json -c cam_imu.yaml # camera–IMU, with its reference: blockEach check is pass, warn or fail.
The grade is RED if any check fails, YELLOW if any warns, GREEN otherwise.
The camera–IMU HTML report uses the same thresholds for its badges.
| Check | id | Pass | Warn | Fail |
|---|---|---|---|---|
| Reprojection median | reproj.median | ≤ 1.41 px | ≤ 2.83 px | above |
| Reprojection P95 | reproj.p95 | ≤ 2.83 px | ≤ 5.66 px | above |
| Reprojection RMS (when no median) | reproj.rms | ≤ 1.41 px | ≤ 2.83 px | above |
| Camera and IMU fit, χ²/dof | fit.cam_chi2_dof, fit.imu_chi2_dof, fit.aux_imu_chi2_dof | ≤ 1.5 | ≤ 3.0 | above |
| Focal length | intr.focal_positive | fx, fy > 0 | — | otherwise |
| Focal aspect | intr.focal_aspect | |fx/fy − 1| ≤ 0.02 | above | — |
| Principal point | intr.principal_point | within 10 % of the image size from the centre | further | — |
| Distortion | intr.distortion_finite | finite | — | not finite |
| Reprojection present | reproj.missing | every camera has errors | — | a camera has none |
| Cameras in the result | result.cameras | at least one | — | none |
| Corner observations per camera (camera-only) | data.observations | ≥ 500 | fewer | — |
| IMU scale (camera–IMU) | imu.dw_scale, imu.da_scale | |diagonal − 1| ≤ 0.05 | above | — |
| Camera–IMU time offset | time.td | |td| ≤ 50 ms | above | — |
| Solver status | solve.ok | true | — | false |
| Iterations | solve.converged | below max_iterations | hit the cap | — |
| Solver step | solve.lm_step | took at least one step | took no step | took no step and failed |
Lever against reference: | ref.lever | ≤ 10 mm | above | — |
Baseline against reference: | ref.baseline | ≤ 5 mm | above | — |
Notes:
- Pixel units: the reprojection thresholds are on the length of each corner’s 2D error, as Kalibr reports it. They are per-axis limits times √2: 1 and 2 px for the median and RMS, 2 and 4 px for the P95.
- χ²/dof: the solve’s cost per degree of freedom. Near 1 means the noise settings match the data.
- Warnings can be real: a raw consumer IMU can have a scale error above 5 %, and a USB camera
a
tdabove 50 ms. Confirm them across recordings. ok: truefrom the solver only means it finished and passed its own gates.verdictnever counts it as proof of a good fit.
3. Check it independently
Section titled “3. Check it independently”mvis verdict --independent-check <kind> records which of these you ran.
Round trip
Section titled “Round trip”mvis roundtrip -c cam_imu.yamlIt needs no ground truth. MVIS applies the calibration to its own bag and calibrates the corrected bag again. PASS needs every one of these on the corrected bag:
| Quantity | Expected after correction | Tolerance |
|---|---|---|
Dw, Da diagonals | 1 | within 0.01 |
| Gyroscope and accelerometer bias | 0 | below 0.02 (rad/s, m/s²) |
| Lever and stereo baseline | unchanged | within 2 mm |
td | 0 | above 2 ms is a warning only, because td can be weakly observable |
The IMU terms must collapse to identity and zero while the geometry stays the same. See Round-trip validation.
Repeatability
Section titled “Repeatability”Calibrate several recordings of one rig with input.bags and compare the results.
A parameter that moves between recordings was not pinned down by the motion in them.
See Camera–IMU calibration.
Another tool
Section titled “Another tool”Run Kalibr or Basalt on the same recordings.
mvis.io.kalibr reads Kalibr’s camchains and results (Kalibr files).
The leaderboard shows MVIS, Kalibr and Basalt side by side.
Digital twin
Section titled “Digital twin”A digital twin replays a real recording’s motion, frame times and corner layout with known parameters. Solving it measures bias against a truth no one gave the solver. See Digital-twin check.
Benchmark metrics
Section titled “Benchmark metrics”The leaderboard and each dataset’s eval page use these numbers. Methodology explains how to read them.
| Metric | Unit | What it is |
|---|---|---|
| Median ± sd (n) | per parameter | Median and standard deviation of one tool’s estimate over the n recordings it solved |
| Difference | per parameter | Difference of two tools’ medians; ◆ when it is larger than the larger of the two spreads |
| Focal length, principal point | px | Camera intrinsics |
| Lever arm | mm | Camera position in the base IMU frame, or auxiliary IMU position |
| Rotation | rad | log(R) rotation vector of an extrinsic |
| Stereo baseline | mm | Distance between two cameras |
td, t_ga | ms | Camera–IMU and gyro-to-accelerometer time offsets |
| Solved | n of N | Recordings a tool solved |
The benchmark datasets have no ground truth. A small spread shows repeatability, not accuracy, and a difference shows that two tools disagree, not which one is right. Reprojection error is not compared across tools, because it measures fit.
Reference values
Section titled “Reference values”Where a recording has published or factory values, the regression suite reports the
percentage difference to them (--gt). These values are compared, never used as a pass or fail gate.
They are kept with the regression suite.
Regression suite
Section titled “Regression suite”We run MVIS over real recordings and digital twins, and gate every change against stored baselines with per-metric tolerances (focal length, distortion, extrinsics, time offsets, readout, reprojection). The runner and its baselines stay with the maintainers, not in the public repository; the commands below are how we run it. See Regression testing for the tests you can run.
python scripts/run_regression.py --mode all --level fast # the gatepython scripts/run_regression.py --only-collection mvis_4imus_3cams/april_01python scripts/run_regression.py --mode all --level all --dry # compare existing results only--level | Runs |
|---|---|
fast | The first recording of each family: the gate for every change |
mid | Three recordings per family |
full | Every recording of every family |
optional | Rows outside the gates, such as camera-model variants |
all | full and optional |
--help prints the current row counts.
--update-baseline rewrites the stored baselines after an intended change; the baseline diff
then shows every number the change moved.
How the suite is built: Regression testing.