Config

Lightweight config system based on a section in the pyproject.toml file.

SbatchArgs ¶

SbatchArgs = dict[str, str | int | float | bool]

A set of sbatch flags, as a mapping from flag name to value.

Returned by:

Reference
- CLI submit
  - merge_sbatch_args
  - sbatch_args_from_args_list
- Config

Used by:

Reference CLI submit

PartialClusterConfig ¶

Per-cluster configuration options.

Returned by:

Reference Config CluvConfig clusters

env ¶

env: dict[str, str] = field(default_factory=dict)

Environment variables to set when running Slurm commands on this cluster.

sbatch_args ¶

sbatch_args: SbatchArgs | list[SbatchArgs] = field(
    default_factory=dict
)

Per-cluster sbatch flags, overriding the global sbatch_args.

A list of flag sets can be used when you have access to more than one allocation on this cluster (for example a def- and an rrg- account). cluv submit <cluster> then submits one job per allocation and keeps the first one that starts, cancelling the others.

results_path ¶

results_path: str | None = None

Path to the results directory for a specific cluster.

datasets_path ¶

datasets_path: str | None = None

Different path where the datasets should be replicated on this cluster.

When None, this defaults to the top-level config's datasets_path.

This folder will be synced from the current cluster to all other clusters at their respective dataset_path.

job_script_path ¶

job_script_path: str | None = None

Path to the job script to use by default on this cluster.

project_dir ¶

project_dir: str | None = None

Path where the project should be cloned on this cluster.

ClusterConfig ¶

Bases: Generic[PathType]

Per-cluster configuration options.

The path fields in this class are by default 'pure' posix paths, to make it explicit that they can't be used on the local filesystem, and to avoid errors like trying to call .open/.mkdir/etc. On the current cluster, when calling current_cluster_config, they are actual Paths.

Returned by:

Reference Config
- CluvConfig get_cluster_config
- current_cluster_config

env ¶

env: dict[str, str]

Environment variables to set when running Slurm commands on this cluster.

sbatch_args ¶

sbatch_args: list[SbatchArgs]

Merged sbatch flags (global defaults overridden by per-cluster values).

There is one entry per allocation configured for this cluster, and always at least one.

results_path ¶

results_path: PathType

Path to the results directory for a specific cluster.

datasets_path ¶

datasets_path: PathType | None

Different path where the datasets should be replicated on this cluster.

When None, this defaults to the top-level config's datasets_path.

This folder will be synced from the current cluster to all other clusters at their respective dataset_path.

job_script_path ¶

job_script_path: PathType | None

Path to the job script to use by default on this cluster.

project_dir ¶

project_dir: PathType | None

Path where the project should be cloned on this cluster.

LocalConfig ¶

Config for using cluv on a local machine (not on a Slurm cluster).

env ¶

env: dict[str, str] = field(default_factory=dict)

Environment variables to set when using cluv on a local machine (not on a Slurm cluster). For example, this can be used to set a fake "$SCRATCH" directory to use when not on a Slurm cluster.

CluvConfig ¶

Bases: BaseModel

Configuration options for Cluv, loaded from the pyproject.toml file.

Returned by:

Reference Config get_cluv_config

Used by:

Reference CLI sync fetch_results

env ¶

env: dict[str, str] = {}

Global environment variables set on all clusters when running Slurm commands.

sbatch_args ¶

sbatch_args: SbatchArgs = {}

Global sbatch flags applied on all clusters.

These are passed directly to sbatch and complement env (which sets SBATCH_* env vars). See [tool.cluv.clusters.<name>.sbatch_args] for per-cluster overrides.

results_path ¶

results_path: str

Default path to the results directory for all clusters (may contain env vars like $SCRATCH).

results_symlink ¶

results_symlink: str = 'logs'

Name of the symlink created in the project directory pointing to results_path.

data_source ¶

data_source: str | None = None

hostname:/path or /local/path of where to get the data from.

When set to hostname:/path, Cluv pulls the dataset from the given remote cluster before pushing it to all target clusters. When set to a plain path (no hostname: prefix), the data is read directly from that local path and pushed to each target cluster without a prior pull step.

datasets_path ¶

datasets_path: str | None = None

Path to a dataset directory, for example, '$SCRATCH/my_dataset'

This folder will be synced from the current cluster to all other clusters at their respective dataset_path.

job_script_path ¶

job_script_path: str | None = None

Default path to the job script to submit when one is not passed explicitly to cluv submit.

This can be overridden for specific clusters in the clusters section, and can also be overridden on the fly by passing a different job script to cluv submit.

project_dir ¶

project_dir: str | None = None

Default path where the project should be cloned on clusters.

clusters ¶

clusters: dict[str, PartialClusterConfig] = {}

Configuration options for each cluster.

The keys are cluster names, and values are configs that override options for that cluster.

get_cluster_config ¶

get_cluster_config(cluster: str) -> ClusterConfig

Returns the cluster config for a specific cluster.

The environment variables as part of paths will not be resolved.

get_cluv_config ¶

get_cluv_config() -> CluvConfig

Get the cluv config, loading it from the pyproject.toml if needed.

has_cluv_config ¶

has_cluv_config(pyproject_path: Path) -> bool

Check if the pyproject.toml contains a cluv config

current_cluster_config ¶

current_cluster_config() -> ClusterConfig[Path] | None

Returns the ClusterConfig of the current cluster, or None if not currently on a cluster.