Evaluation & metrics

How to tell whether a calibration is good. The same metrics back the leaderboard, so a local run and a published entry are comparable.

Read the report first

Every run writes a self-contained HTML report. The section to read before the error numbers is calibration data quality: motion-excitation metrics with letter grades, rotation-axis spread, angular-velocity timeline, and a feature-coverage heatmap with convex hull.

Reprojection error is not accuracy

It measures how well the solution explains the data it was fitted to. A parameter the motion never excited settles near its initial value without raising the residual. A low error on a poorly-excited sequence means very little.

Metrics

Metric	Units	Against	Notes
Reprojection median / P95	px	own observations	Robust statistics, not mean — P95 is where model mismatch shows
Extrinsic rotation error	deg	a reference	Geodesic angle
Extrinsic translation error	cm	a reference	Includes the camera–IMU lever arm
Stereo baseline	mm	a reference	The most stable cross-check on a rig
Time-offset error	ms	a reference	Per camera and auxiliary IMU
Runtime	s	—	Wall-clock, hardware reported
Convergence rate	%	repeated runs	Successful / total

Read convergence first: excellent errors at a 60% convergence rate is not a better toolbox than one that always converges.

Ground truth, where it exists

Per-dataset ground-truth YAMLs live under examples/regression/gt/. The dashboard annotates each parameter with its deviation — Δ +0.27% GT=190.97 — colour-coded by accuracy band. Version 3 results match the published TUM-VI and Intel T265 calibrations to under 0.6% on focals.

Round-trip: the strongest check without ground truth

camcalib roundtrip -c cam_imu.yaml

PASS requires IMU intrinsics to collapse to identity on the corrected bag and extrinsics and stereo baseline to be preserved. Two conditions pulling in opposite directions is what makes it hard to satisfy by accident. See Round-trip validation.

Repeatability across sequences

The most practical field check. Run a batch sweep and compare: parameters that move between sequences were not constrained by the motion in them. The regression harness reports this directly as mean ± stddev of intrinsics, distortion, and baseline across a collection.

Regression harness

Manifest-driven batch runs with per-run HTML reports, a colour-coded index, and a cross-collection repeatability summary:

python3 scripts/run_regression_tests.py --level benchmark --parallel 8
python3 scripts/run_quick_regression.py            # ~6 runs, ~10 min
python3 scripts/run_regression_tests.py --only tum_vi_standard

Dashboards land at results/regression/{index,summary}.html. Details in doc/regression_tests.md.

Baselines

python3 scripts/compare_baseline.py --verbose
python3 scripts/save_baseline.py --name baseline_2026_04_20

Saving a baseline generates a comparison against the previous one: per-dataset reprojection errors, timing, camera parameters (mean ± stddev), extrinsics, and rolling-shutter readout, with ground-truth values where available. This is how a change is shown not to have regressed anything.

Timing prediction

A physics-informed Ridge model (v2, R² = 0.97) trained on accumulated runs predicts runtime, with a linear v1 fallback. It retrains after each regression run and baseline save, and accounts for core contention when predicting parallel runs.

Current validated state

228 runs across 65 dataset families, zero failures.