Regression testing
Use this page before you change MVIS code. It lists the test tiers you can run from a clone of github.com/openmvis/openmvis and tells you which one to run when.
Set up
Section titled “Set up”git clone https://github.com/openmvis/openmviscd openmvispython -m venv .venv && source .venv/bin/activate # Python 3.11 or 3.12, Linux or macOSpip install -e ".[apriltag,aruco,ros1,dev]"The dev extra installs pytest. See Installation
for the other extras.
The four tiers
Section titled “The four tiers”Run the smallest tier that covers your change. Each one is a single command from the repository root.
| Tier | Command | What it runs | Time | When |
|---|---|---|---|---|
| Unit | scripts/test.sh unit | every test except the slow and integration ones | about 4 min | while coding |
| Solver | scripts/test.sh solver | five real solves: a simulated camera–IMU solve, the t_ga term end to end, two checks on one digital twin, and a check that the fast linearised IMU factor matches the exact one | about 1 min | with unit; CI runs it |
| Integration | scripts/test.sh integration | digital twins, and a simulated camera whose frames are stamped by polling the sensor clock, through the full solver | about 3 min | before you commit a solver change |
| Full | scripts/test.sh full | everything in tests/ | about 20 min | before a merge |
t_ga: the time offset between the gyroscope and the accelerometer of one IMU. See Why t_ga.- Times are approximate and depend on your machine.
- What
scripts/test.shdoes: it pins the linear-algebra library (BLAS) to one thread, so results repeat. It tests thesrc/of this checkout, not an installed copy. - Extra arguments go to pytest:
scripts/test.sh unit -k imu. - Interpreter: set
PYTHONto choose it.
What CI runs
Section titled “What CI runs”- When: on every pull request and every push to
main. - What: the unit and solver tiers only.
- What it skips: the slow and integration tests. Run
scripts/test.sh fullyourself before you ask for a merge.
Tests that need recordings
Section titled “Tests that need recordings”Some tests read real recordings. They skip when the data is missing, so a clean clone passes without any download.
To run them:
- Log in with
huggingface-cli login, or setHF_TOKEN. The VI-Rig sets are gated: accept their terms on the dataset page first. - Pick a folder for the data and set
MVIS_DATASETSto it. Without it, MVIS uses~/Datasets. - Download what you need with
mvis datasets download. Usemvis datasets listto see the catalogue, and--dataset <key>to fetch one dataset.
huggingface-cli loginexport MVIS_DATASETS=/path/to/datasetsmvis datasets listmvis datasets download --dataset <key>scripts/test.sh fullSee Datasets for what each set contains and its licence.
Digital-twin tests
Section titled “Digital-twin tests”A digital twin is a simulated copy of a real recording. The integration tier solves twins of two real stereo camera–IMU recordings, without images:
| Twin | Covers |
|---|---|
twin_gs | global-shutter cameras, 470 Hz IMU, t_ga +5.3 ms |
twin_rs | rolling-shutter cameras with negative readout (rows read bottom to top), 200 Hz IMU, t_ga −7.4 ms |
How a twin works:
- The truth: a real solve, moved by a fixed random step so that no real
device calibration ships in the test files. Focal length moves by up to 1 %,
the principal point by 4 px, other intrinsics by 5 %, the extrinsic rotation by
10 mrad and the lever arm by 5 mm per axis,
td(the time offset between the camera and IMU clocks) by 1 ms andt_gaby 0.5 ms. - What it keeps: the real frame times, corner ids, IMU timestamps and solve settings.
- The precise check: the solve without noise. Any error left there is bias in the estimator.
- The limits: set just above the measured errors, so a failure means something changed.
- Iteration check: each test also checks that the optimiser iterated. A
result flagged
okalone does not show that it solved.
The noise is Gaussian, and a twin has no lens or timing model errors. So the
twin tests check the estimator, not a real recording. The fixtures are in
tests/data/twins/. Making a new fixture needs the recording; running the tests
needs only the fixture.
To simulate your own rig, see Digital-twin check.
Where new tests go
Section titled “Where new tests go”- Place:
tests/<area>/test_<topic>.py, one folder per area:camera,detection,calibration,cam_imu,simulation,config,io,mvisdb,datasets,cli,recalibandintegration. - Helpers: shared helpers go in
tests/helpers/. A test module does not import from another test module. - Library code belongs in
src/mvis. Tests import fromscripts/only to test a script’s own command line. - Do not refactor
tests/_reference/. It is a frozen copy of earlier code that the speed-up tests compare against bit for bit. - Slow tests:
tests/slow_tests.txtlists the tests that take over 5 s, andtests/conftest.pymarks themslow. Add a new slow test to that list by hand; no marker is needed in the test file.
Beyond the test tiers
Section titled “Beyond the test tiers”We also run two larger suites: real benchmark recordings, and digital twins of them, each compared parameter by parameter with earlier approved results. They need the full dataset collection and many hours of compute, so they are not part of the tiers above. The same recordings, solved with MVIS, Kalibr and Basalt, make the leaderboard; Methodology explains how we run them and Evaluation & metrics defines the metrics.
Related pages
Section titled “Related pages”- CONTRIBUTING.md: pull-request rules and the licence header for new files
- Governance