sync

sync ¶

sync(
    clusters: list[str] | None = None,
    sync_datasets: bool = True,
) -> list[Remote]

Synchronizes the current project across clusters.

Synchronizes code across all clusters.
Gathers results on the "main" cluster (mila)
Does uv sync that cluster as well
- (Important so that jobs can be run in OFFLINE mode)

Parameters:

Name	Type	Description	Default
`clusters`	`list[str] \| None`	List of SSH hostnames of the target clusters. If empty, will attempt to sync with all clusters in the config that we have an active SSH connection to.	`None`
`sync_datasets`	`bool`	Whether to pull/push datasets from/to `data_source` as part of the sync.	`True`

Returns:

Type	Description
`list[Remote]`	A list of Remote objects corresponding to the clusters that were synced with.

get_cluster_to_remote ¶

get_cluster_to_remote(
    cluster: Literal["first"] | str | list[str] | None,
) -> dict[str, Remote | None]

Resolves cluster name(s) to Remotes, logging in to any that aren't already connected.

Always includes the current cluster (mapped to None, meaning "run locally") if we're on one. When cluster is "first" or None, returns every cluster we have (or can get) an active connection to, plus the current cluster.

get_active_remotes ¶

get_active_remotes() -> list[Remote]

Returns the Remotes for each cluster which has an active SSH connection.

Disabled clusters (see cluv disable) are excluded. Note that this can include more than one cluster from the same CLUSTERS_SHARING_A_FILESYSTEM group (e.g. both trillium and trillium-gpu): they're genuinely different Slurm clusters to submit jobs to, even though sync() only syncs the underlying (shared) checkout once.

sync_common_part ¶

sync_common_part(
    remotes: list[Remote], sync_datasets: bool = True
) -> None

Sync steps that only need to happen once, regardless of how many clusters we're syncing or submitting to: push the local commit, and pull the dataset from its source cluster if needed.

sync_per_cluster_part ¶

sync_per_cluster_part(
    cluster_remote: Remote | None,
    sync_datasets: bool = True,
) -> list[Path]

Sync steps specific to one cluster: install uv, clone/update the project, uv sync, fetch back new results, and push datasets to it if needed.

Does nothing (and returns an empty list) when cluster_remote is None (the current cluster), since there's nothing to sync to it.

expandvars ¶

expandvars(
    remote: Remote, path: str | PurePosixPath
) -> PurePosixPath

Same idea as os.path.expandvars, but for a path on a remote machine. Just uses echo.

clone_project ¶

clone_project(
    remote: Remote,
    project_path: PurePosixPath,
    project_state: ProjectStateOnCluster,
)

Setup the project repo on all the remote clusters.

New idea: - Assume GitHub. Push to GitHub if needed. Clone from github on the remotes. - Worry about authentication later, just raise an error if need be for now.

fetch_results ¶

fetch_results(
    remote: Remote,
    config: CluvConfig,
    project_state: ProjectStateOnCluster,
) -> list[Path]

Fetches results from a remote cluster to local using rsync via the results symlink.

Returns the list of newly-synced run directories (those that did not exist locally before the rsync ran). Also updates project_state.last_fetch_watermark (see cluv clean).

create_results_dir_with_symlink_to_scratch ¶

create_results_dir_with_symlink_to_scratch(
    remote: Remote,
    project_dir: PurePosixPath,
    results_symlink: str,
    results_path: PurePosixPath,
)

On the remote, create results_path and symlink project/ -> results_path.

results_path may contain env vars (e.g. $SCRATCH); they are resolved via the remote login shell.

remote_test ¶

remote_test(
    flag: Literal["-d", "-e", "-L"],
    path: str | PurePosixPath,
    remote: Remote,
) -> bool

Returns True if test {flag} {path} succeeds on the remote.