Skip to content

Env vars

SLURM_JOB_ID module-attribute #

SLURM_JOB_ID: int | None = (
    int(environ["SLURM_JOB_ID"])
    if "SLURM_JOB_ID" in environ
    else None
)

The value of the 'SLURM_JOB_ID' environment variable.

See https://slurm.schedmd.com/sbatch.html#OPT_SLURM_JOB_ID.

SLURM_TMPDIR module-attribute #

SLURM_TMPDIR: Path | None = (
    Path(environ["SLURM_TMPDIR"])
    if "SLURM_TMPDIR" in environ
    else (
        tmp
        if SLURM_JOB_ID is not None and exists()
        else None
    )
)

The SLURM temporary directory, the fastest storage available.

  • Extract your dataset to this directory at the start of your job.
  • Remember to move any files created here to $SCRATCH since everything gets deleted at the end of the job.

See https://docs.mila.quebec/Information.html#slurm-tmpdir for more information.

SCRATCH module-attribute #

SCRATCH = (
    Path(environ["SCRATCH"])
    if "SCRATCH" in environ
    else None
)

Network directory where temporary logs / checkpoints / custom datasets should be saved.

Note that this is temporary storage. Files that you wish to be saved long-term should be saved to the ARCHIVE directory.

See https://docs.mila.quebec/Information.html#scratch for more information.

ARCHIVE module-attribute #

ARCHIVE = (
    Path(environ["ARCHIVE"])
    if "ARCHIVE" in environ
    else None
)

Network directory for long-term storage. Only accessible from the login or cpu-only compute nodes.

See https://docs.mila.quebec/Information.html#archive for more information.

NETWORK_DIR module-attribute #

NETWORK_DIR = (
    Path(environ["NETWORK_DIR"])
    if "NETWORK_DIR" in environ
    else _network_dir if exists() else None
)

The (read-only) network directory that contains datasets/weights/etc.

todo: adapt this for the DRAC clusters.

When running outside of the mila/DRAC clusters, this will be None, but can be mocked by setting the NETWORK_DIR environment variable.

REPO_ROOTDIR module-attribute #

REPO_ROOTDIR = parent

The root directory of this repository on this machine.

DATA_DIR module-attribute #

DATA_DIR = Path(
    get(
        "DATA_DIR",
        SLURM_TMPDIR or SCRATCH or REPO_ROOTDIR / "data",
    )
)

Local Directory where datasets should be extracted on this machine.

torchvision_dir module-attribute #

torchvision_dir: Path | None = None

Network directory with torchvision datasets.

NUM_WORKERS module-attribute #

NUM_WORKERS = int(
    get(
        "SLURM_CPUS_PER_TASK",
        get(
            "SLURM_CPUS_ON_NODE",
            (
                len(sched_getaffinity(0))
                if hasattr(os, "sched_getaffinity")
                else cpu_count()
            ),
        ),
    )
)

Default number of workers to be used by dataloaders, based on the number of CPUs and/or tasks.

get_constant #

get_constant(*names: str)

Resolver for Hydra to get the value of a constant in this file.