Using Cluv with Hydra

The Cluv Hydra Launcher lets you run Hydra multi-run sweeps directly on remote Slurm clusters, using the same pyproject.toml-based config that drives cluv submit.

It is a drop-in replacement for the Submitit launcher plugin — the same gpus_per_node, cpus_per_task, mem_gb, timeout_min, etc. parameters all work as-is.

What it adds on top of Submitit:

Allows using remote clusters: Cluv allows you to launch jobs on the current cluster as well as remote clusters.
Automatic sync: the project is synced to the target cluster before submission (via cluv sync).
Automatic result fetch: results are rsynced back locally once jobs finish.
Cluster selection: set cluster: mila (or any cluster in your config) to pick the target. Default is 'first' to use the first cluster that runs the job.

${cluv:...} resolver: access job information (e.g. results_path) from Hydra configs. (1)

This is similar in spirit to the JobEnvironment class of submitit.

1. Installation

Add the hydra extra when installing cluv:

uv add git+https://github.com/mila-iqia/cluv --extra hydra

Cluv isn't published on PyPI yet. Once it is, you will be able to just uv add cluv[hydra].

2. Configure your project

Your pyproject.toml needs a [tool.cluv] section with at least a results_path and the clusters you want to target. A minimal setup can be obtained by running cluv init.

Take a look at the pyproject.toml file of this example:

pyproject.toml

# Where to store job results by default.
results_path = "$SCRATCH/logs/hydra_example"
# On clusters, Cluv creates a symlink (a shortcut) in your project folder to the results_path dir.
# This makes it easier to keep your project in $HOME and to see the results which are on $SCRATCH.
results_symlink = "logs"
# Where to read the data from when synchronizing data to all clusters.
data_source = "mila:/network/datasets/cifar10.var/cifar10_torchvision"
# Where the dataset should be replicated on all clusters.
datasets_path = "$SCRATCH/datasets/cifar10"

[tool.cluv.env]
# Assume that compute nodes don't have internet access by default. Override below when they do.
UV_OFFLINE="1"
WANDB_MODE="offline"

[tool.cluv.sbatch_args]
# Environment variables applied when using Slurm commands on all clusters.
time = "3:00:00"
requeue = true


###  --------------   Clusters Config   --------------  ###

[tool.cluv.clusters.mila]
# Overrides specific to the Mila cluster.
env = { UV_OFFLINE = "0", WANDB_MODE = "online" }
results_path = "$SCRATCH/logs/hydra_example"

[tool.cluv.clusters.tamia]

[tool.cluv.clusters.killarney]
# For example, you might not have a $SCRATCH on Killarney. This can be overwritten here.
datasets_path = "$HOME/datasets/cifar10"
results_path = "$HOME/logs/hydra_example"

[tool.cluv.clusters.vulcan]

[tool.cluv.clusters.rorqual]
[tool.cluv.clusters.rorqual.sbatch_args]
account = "rrg-bengioy-ad"

[tool.cluv.clusters.fir]
env = {UV_OFFLINE="0", WANDB_MODE="online"}

[tool.cluv.clusters.nibi]
env = {UV_OFFLINE="0", WANDB_MODE="online"}
[tool.cluv.clusters.nibi.sbatch_args]
account = "rrg-bengioy-ad"

[tool.cluv.clusters.trillium.sbatch_args]
account = "rrg-bengioy-ad"

[tool.cluv.clusters.trillium-gpu.sbatch_args]
account = "rrg-bengioy-ad"

[tool.cluv.clusters.narval.sbatch_args]
# Mila doesn't have an allocation on Narval anymore.
account = "def-bengioy"

See config reference for all available fields.

3. Add a job script

The launcher submits jobs using a shell script (just like cluv submit). The script receives the Python command as positional arguments via $@:

scripts/job.sh

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=0:05:00

# Note: --output is set by cluv. No worries there.

# Run the job command passed as an argument when submitting the job ('python main.py' for example)
echo "Running command: $@"
srun uv run "$@"

Tip

The --output flag is injected by the launcher, so you don't need it in the script.

4. Add the launcher config

Create a Hydra config file that selects the Cluv launcher. This is typically placed in configs/launcher/cluv.yaml so it can be activated with +launcher=cluv on the command line:

configs/launcher/cluv.yaml

# @package _global_
defaults:
  - override /hydra/launcher: cluv_launcher

hydra:
  mode: MULTIRUN
  run:
    # Output directory. Normally would be {name}/{now:%Y-%m-%d/%H-%M-%S}} but
    # here we instead use the directory created by cluv based on the config in pyproject.toml
    # and the cluster name and job id. This is typically "{cluster.results_path}/{cluster_name}_{job_id}",
    # See `cluv.job.current_run_info` for more details.
    # TODO: Weird that we have to supply a default value.
    # Hydra seems to want to create a directory locally when launching a sweep.
    dir: ${cluv:results_path,/tmp/cluv_logs/${now:%Y-%m-%d}/${now:%H-%M-%S}}
  sweep:
    dir: ${cluv:results_path,/tmp/cluv_logs/${now:%Y-%m-%d}/${now:%H-%M-%S}}
    subdir: ${hydra.job.num}

  launcher:
    ## NEW ARGUMENTS for cluv:
    cluster: mila
    job_script: scripts/job.sh
    # chunking: true  # (Coming soon: Automatically chunk jobs into shorter chunks)
    # vram_gb: 10  # (Coming soon: Automatically pack multiple runs per GPU)
    ## Usual submitit arguments:
    stderr_to_stdout: true
    timeout_min: 60
    gpus_per_node: 1
    cpus_per_task: 2
    mem_gb: 16

cluster: first

Use cluster: first to automatically pick the first cluster that already has an active SSH connection (i.e. the first result of cluv status). This avoids hardcoding a cluster name.

Migrating from the Submitit launcher

If you already have a configs/launcher/submitit.yaml, switching to Cluv only requires two changes:

# Before:
defaults:
  - override /hydra/launcher: submitit_slurm

# After:
defaults:
  - override /hydra/launcher: cluv_launcher

hydra:
  launcher:
    cluster: mila     # add this
    # everything else stays the same

5. Run a sweep

First, make sure you have active SSH connections:

cluv login

Then launch your sweep the normal Hydra way, activating the launcher with +launcher=cluv:

python main.py -m +launcher=cluv lr=0.01,0.001 seed=1,2,3

The launcher will:

Sync your project to the target cluster (cluv sync).
Submit one sbatch job per config combination.
Monitor jobs until all complete.
Rsync results back to your local results_symlink directory.

6. The `${cluv:...}` resolver

The launcher registers a custom OmegaConf resolver so Hydra configs can read live cluv job info:

${cluv:<attribute>,<default>}

Attribute	Description
`results_path`	The resolved results path for the current job
`cluster`	Name of the cluster the job is running on
`run_id`	Unique run identifier (`{cluster}_{job_id}_{task_id}`)

Example — point Hydra's output dir to the cluv-managed results directory:

hydra:
  sweep:
    dir: ${cluv:results_path,/tmp/cluv_logs/${now:%Y-%m-%d}/${now:%H-%M-%S}}
    subdir: ${hydra.job.num}

The second argument (after the comma) is the default value, used when the job is not running inside Slurm — for example, during a local dry-run.

7. Reading cluster info inside your script

Use cluv.job.current_run_info() to access cluster-specific settings at runtime:

import cluv.job
import cluv.config

run_info = cluv.job.current_run_info()

if run_info:
    # Running on Slurm — use per-cluster config
    datasets_path = run_info.cluster_config.datasets_path
else:
    # Running locally
    datasets_path = cluv.config.get_cluv_config().datasets_path

current_run_info() returns None when the script is not running inside a Slurm job, so this pattern works both locally and on the cluster without any changes.

Full example

See examples/hydra_example/ for a complete working example with CIFAR-10, Weights & Biases logging, and multi-cluster config.