Env vars
SLURM_JOB_ID
module-attribute
#
The value of the 'SLURM_JOB_ID' environment variable.
SLURM_TMPDIR
module-attribute
#
SLURM_TMPDIR: Path | None = (
Path(environ["SLURM_TMPDIR"])
if "SLURM_TMPDIR" in environ
else (
tmp
if SLURM_JOB_ID is not None and exists()
else None
)
)
The SLURM temporary directory, the fastest storage available.
- Extract your dataset to this directory at the start of your job.
- Remember to move any files created here to $SCRATCH since everything gets deleted at the end of the job.
See https://docs.mila.quebec/Information.html#slurm-tmpdir for more information.
SCRATCH
module-attribute
#
Network directory where temporary logs / checkpoints / custom datasets should be saved.
Note that this is temporary storage. Files that you wish to be saved long-term should be saved to the ARCHIVE
directory.
See https://docs.mila.quebec/Information.html#scratch for more information.
ARCHIVE
module-attribute
#
Network directory for long-term storage. Only accessible from the login or cpu-only compute nodes.
See https://docs.mila.quebec/Information.html#archive for more information.
NETWORK_DIR
module-attribute
#
NETWORK_DIR = (
Path(environ["NETWORK_DIR"])
if "NETWORK_DIR" in environ
else _network_dir if exists() else None
)
The (read-only) network directory that contains datasets/weights/etc.
todo: adapt this for the DRAC clusters.
When running outside of the mila/DRAC clusters, this will be None
, but can be mocked by setting the NETWORK_DIR
environment variable.
REPO_ROOTDIR
module-attribute
#
The root directory of this repository on this machine.
DATA_DIR
module-attribute
#
DATA_DIR = Path(
get(
"DATA_DIR",
SLURM_TMPDIR or SCRATCH or REPO_ROOTDIR / "data",
)
)
Local Directory where datasets should be extracted on this machine.
torchvision_dir
module-attribute
#
torchvision_dir: Path | None = None
Network directory with torchvision datasets.