Writing a job script

cluv submit doesn't have a special format for job scripts: it's a plain bash script, with #SBATCH directives, submitted as-is via sbatch. cluv just adds a few conventions on top so it can fill in cluster-specific details.

Job script conventions

On the remote cluster, cluv submit invokes the job script roughly as:

sbatch --chdir=<remote_project_dir> [sbatch-args] <job_script> [program_args...]

• The script runs with its working directory set to the project root on the target cluster (--chdir), so relative paths inside the script resolve from there.

• Anything passed after -- on the cluv submit command line is forwarded as positional arguments ($@) to the script, which just forwards them to uv run.

cluv submit mila scripts/job.sh -- python main.py --lr 0.01 runs uv run python main.py --lr 0.01 on the cluster. In case of a custom job script, make sure to forward the arguments to your program, e.g. "$@" in bash.

Cluv job scripts

By default, the cluv init command tries to create two job scripts in the scripts folder: job.sh and safe_job.sh.

job.sh

The simplest possible job script where it just forwards its arguments to uv run. It's the default value for job_script_path in your config (see "Configuring job submission" page), when no script is used in the CLI.

scripts/job.sh

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=0:05:00

# Run the job command passed as an argument when submitting the job ('python main.py' for example)
echo "Running command: $@"
srun uv run "$@"

safe_job.sh

Guarding against the project changing under a queued job.

The simple script above runs directly out of the synced project directory. That's fine for short, immediate jobs, but if the job sits in the Slurm queue for a while, a later cluv sync (or another cluv submit) could change the checked-out commit before the job actually starts running.

The safer pattern is to clone the project into $SLURM_TMPDIR and explicitly check out $GIT_COMMIT, so the job always runs the exact commit it was submitted with, independent of what's currently synced in the project root:

scripts/safe_job.sh

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=0:05:00

project_name="cluv"  # to be replaced with the user's project name.
project_root="$HOME/repos/$project_name" # to be replaced with the path to the user's project in their $HOME.
results_path="logs" # to be replaced with the path to the results path name. (--output flag above too)

echo "GIT_COMMIT=${GIT_COMMIT:?GIT_COMMIT is not set. Use 'cluv submit' to submit this job script.}"
# Setup the repo in $SLURM_TMPDIR, so the code can change in the project without affecting the job.
project_root_in_tmpdir="$SLURM_TMPDIR/$project_name"
echo "Cloning the project and setting up the virtual environment in $project_root_in_tmpdir"

srun --ntasks-per-node=1 --ntasks=$SLURM_JOB_NUM_NODES bash -e <<END
    cd $SLURM_TMPDIR
    echo "Cloning the project from $project_root to $SLURM_TMPDIR"
    set -x  # show commands as they are executed (for debugging).

    git clone $project_root  # clone the project from $HOME to $SLURM_TMPDIR
    cd $SLURM_TMPDIR/$project_name
    git checkout --detach $GIT_COMMIT
    # Copy the virtualenv (seems necessary for some clusters in offline mode).
    cp -r $project_root/.venv $SLURM_TMPDIR/$project_name/.venv
    uv sync

    # Copy any existing results from $SCRATCH to the project root.
    mkdir -p $project_root_in_tmpdir/$results_path
    if [ -d "$project_root/$results_path/$SLURM_JOB_ID" ]; then
        rsync --update --recursive $project_root/$results_path/$SLURM_JOB_ID $project_root_in_tmpdir/$results_path/
    fi
END

# Run the actual job command passed as an argument ('python main.py' for example)
echo "Running command: 'uv run $@' in $project_root_in_tmpdir"
srun uv --directory=$project_root_in_tmpdir run "$@"

# Copy results (if any) from the local storage back to the results dir (eg in $SCRATCH)
echo "Copying logs from $project_root_in_tmpdir/$results_path to $project_root/$results_path"
if [ -d "$project_root_in_tmpdir/$results_path/$SLURM_JOB_ID" ]; then
    srun --ntasks-per-node=1 --ntasks=$SLURM_JOB_NUM_NODES \
        rsync --update --recursive $project_root_in_tmpdir/$results_path/$SLURM_JOB_ID $project_root/$results_path/
fi

This variant also copies any existing results for $SLURM_JOB_ID into $SLURM_TMPDIR before running (in case of requeue) and rsyncs them back to results_path afterwards, matching the {results_path}/{cluster}_%j/ layout that SBATCH_OUTPUT uses.