Skip to content

Regression testing

Use this page before you change MVIS code. It lists the test tiers you can run from a clone of github.com/openmvis/openmvis and tells you which one to run when.

Terminal window
git clone https://github.com/openmvis/openmvis
cd openmvis
python -m venv .venv && source .venv/bin/activate # Python 3.11 or 3.12, Linux or macOS
pip install -e ".[apriltag,aruco,ros1,dev]"

The dev extra installs pytest. See Installation for the other extras.

Run the smallest tier that covers your change. Each one is a single command from the repository root.

TierCommandWhat it runsTimeWhen
Unitscripts/test.sh unitevery test except the slow and integration onesabout 4 minwhile coding
Solverscripts/test.sh solverfive real solves: a simulated camera–IMU solve, the t_ga term end to end, two checks on one digital twin, and a check that the fast linearised IMU factor matches the exact oneabout 1 minwith unit; CI runs it
Integrationscripts/test.sh integrationdigital twins, and a simulated camera whose frames are stamped by polling the sensor clock, through the full solverabout 3 minbefore you commit a solver change
Fullscripts/test.sh fulleverything in tests/about 20 minbefore a merge
  • t_ga: the time offset between the gyroscope and the accelerometer of one IMU. See Why t_ga.
  • Times are approximate and depend on your machine.
  • What scripts/test.sh does: it pins the linear-algebra library (BLAS) to one thread, so results repeat. It tests the src/ of this checkout, not an installed copy.
  • Extra arguments go to pytest: scripts/test.sh unit -k imu.
  • Interpreter: set PYTHON to choose it.
  • When: on every pull request and every push to main.
  • What: the unit and solver tiers only.
  • What it skips: the slow and integration tests. Run scripts/test.sh full yourself before you ask for a merge.

Some tests read real recordings. They skip when the data is missing, so a clean clone passes without any download.

To run them:

  1. Log in with huggingface-cli login, or set HF_TOKEN. The VI-Rig sets are gated: accept their terms on the dataset page first.
  2. Pick a folder for the data and set MVIS_DATASETS to it. Without it, MVIS uses ~/Datasets.
  3. Download what you need with mvis datasets download. Use mvis datasets list to see the catalogue, and --dataset <key> to fetch one dataset.
Terminal window
huggingface-cli login
export MVIS_DATASETS=/path/to/datasets
mvis datasets list
mvis datasets download --dataset <key>
scripts/test.sh full

See Datasets for what each set contains and its licence.

A digital twin is a simulated copy of a real recording. The integration tier solves twins of two real stereo camera–IMU recordings, without images:

TwinCovers
twin_gsglobal-shutter cameras, 470 Hz IMU, t_ga +5.3 ms
twin_rsrolling-shutter cameras with negative readout (rows read bottom to top), 200 Hz IMU, t_ga −7.4 ms

How a twin works:

  • The truth: a real solve, moved by a fixed random step so that no real device calibration ships in the test files. Focal length moves by up to 1 %, the principal point by 4 px, other intrinsics by 5 %, the extrinsic rotation by 10 mrad and the lever arm by 5 mm per axis, td (the time offset between the camera and IMU clocks) by 1 ms and t_ga by 0.5 ms.
  • What it keeps: the real frame times, corner ids, IMU timestamps and solve settings.
  • The precise check: the solve without noise. Any error left there is bias in the estimator.
  • The limits: set just above the measured errors, so a failure means something changed.
  • Iteration check: each test also checks that the optimiser iterated. A result flagged ok alone does not show that it solved.

The noise is Gaussian, and a twin has no lens or timing model errors. So the twin tests check the estimator, not a real recording. The fixtures are in tests/data/twins/. Making a new fixture needs the recording; running the tests needs only the fixture.

To simulate your own rig, see Digital-twin check.

  • Place: tests/<area>/test_<topic>.py, one folder per area: camera, detection, calibration, cam_imu, simulation, config, io, mvisdb, datasets, cli, recalib and integration.
  • Helpers: shared helpers go in tests/helpers/. A test module does not import from another test module.
  • Library code belongs in src/mvis. Tests import from scripts/ only to test a script’s own command line.
  • Do not refactor tests/_reference/. It is a frozen copy of earlier code that the speed-up tests compare against bit for bit.
  • Slow tests: tests/slow_tests.txt lists the tests that take over 5 s, and tests/conftest.py marks them slow. Add a new slow test to that list by hand; no marker is needed in the test file.

We also run two larger suites: real benchmark recordings, and digital twins of them, each compared parameter by parameter with earlier approved results. They need the full dataset collection and many hours of compute, so they are not part of the tiers above. The same recordings, solved with MVIS, Kalibr and Basalt, make the leaderboard; Methodology explains how we run them and Evaluation & metrics defines the metrics.