Skip to content

Automated Testing#

Tests are a vital part of any good codebase, especially in Machine Learning. They make it easier to explore and try out new ideas, by giving you the security that your codebase works as intended.

This template comes with some easy-to-use test suites as well as some pre-configured GitHub Actions workflows to run them:

  • Unit tests: quick to run and check small functions / modules / classes.
  • Regression tests: check that your code is reproducible and to let you know if something changed while you were developing your code.
  • integration tests: run your code end-to-end to make sure that all the individually-tested components work together as expected.
  • GitHub Actions runs all these tests before you merge your code.

Automated testing on SLURM clusters with GitHub CI#

🔥 NOTE: This is a feature that is entirely unique to this template! 🔥

This template runs all the above-mentioned tests on an actual Compute Node of the Mila cluster automatically. Assuming that you have access to the Mila / DRAC or other Slurm clusters, all you need to do is to setup a local self-hosted GitHub runner for your fork of this repository, launch it on your local machine with access to a Slurm cluster, and voila: Your code will now be tested on an ACTUAL slurm cluster whenever you push or update a PR in your project GitHub repository.

Detailed instructions on how to set this up in your project will be added soon.

Test-suites#

Unit testing in this template is done with pytest.

To run tests, simply use pytest on the command-line. You may want to add some useful flags like pytest -x -v. See the pytest docs for more info.

The built-in tests cover the following:

  • For each datamodule config, for each data split
    • test that the first batch is always the same
  • For each algorithm config, for all compatible network / datamodule config combinations:
    • initialization is deterministic & reproducibile;
    • forward pass is deterministic & reproducibile;
    • backward pass is deterministic & reproducibile;

Take a look at project.algorithms.testsuites.algorithm_tests to see the included base tests for algorithms.

If you use Visual Studio Code, you may want to look into adding the "test explorer" tab to your editor. Then, you'll be able to see and debug the tests using the GUI.

Unit tests#

pytest -x -v

Regression Tests#

We use pytest-regressions to test that code changes don't break things.

  • --gen-missing: Use this flag when you might be missing some of the regression files (for example on the first test run).
  • --regen-all: Use this when you want to intentionally re-create the regression files. This should hopefully not be used often!

First run#

On the first run, you might want to run test with the --gen-missing files, like so:

pytest --regen-all

integration-tests#

To run slower integration tests, use the following:

pytest -x -v --slow

Continuous Integration#