milabench.pack

This module defines the Package class from which new benchmarks inherit.

A Package must define three methods corresponding to milabench’s three main commands: install, prepare and run. The Package class defines good default behavior.

class milabench.pack.BasePackage(config, core=None)

Base package, with no behavior defined for install/prepare/run.

config: The configuration dictionary as defined in the benchmark config for the benchmark.

pack_path: The path to the package file (same as config["definition"])

dirs

A Namespace object with important paths:

code: The code directory for this benchmark
venv: The virtual environment for this benchmark
data: The data directory (shared)
runs: The runs directory (shared)

async checked_install()

Entry method to install the benchmark.

Check if the benchmark is installed.
install()

conda_install(*args, **kwargs): Install a package using conda.

execute(*args, cwd=None, env={}, external=False, **kwargs)

Run a command in the virtual environment.

Unless specified otherwise, the command is run with self.dirs.code as the cwd.

Parameters:

args – The arguments to the command
cwd – The cwd to use (defaults to self.dirs.code)

make_env(): Return a dict of environment variables to use for prepare/run.

async pip_install(*args, build_isolation=False, use_uv_override=None, **kwargs)

Install a package in the virtual environment.

The arguments are given to pip install verbatim, so you can e.g. do self.pip_install("-r", filename) to install a list of requirements.

async python(*args, **kwargs)

Run a Python script.

Equivalent to:

self.execute("python", *args, **kwargs)

async voir(script, args=(), wrapper=[], cwd=None, **kwargs)

Launch a script using voir.

This runs:

voir {*voirargs} {self.dirs.code / self.main_script} {*args}

Using self.dirs.code as the current working directory.

Note

stderr is piped to stdout in the process

Parameters:

args – A list of arguments to the program.
voirargs – A list of arguments to voir.
env – Environment variables to set for the process.

Returns:

A subprocess.Popen instance representing the running process.

class milabench.pack.Package(config, core=None)

Package class with default behaviors for install/prepare/run.

See:

install()
prepare()
run()

async install()

Install the benchmark.

By default, this installs the requirements file pointed to by the instance or class attribute self.requirements_file, which is set to "requirements.txt" by default. That path is understood to be relative to self.dirs.code. In other words, if self.dirs.code == /blah and self.requirements_file == "requirements.txt", self.install() executes:

pip install -r /blah/requirements.txt``

Note

The main method milabench install calls is checked_install() which takes care of checking if the install already occurred, copying over the manifest’s contents to self.dirs.code, installing milabench in the venv, and then calling this method.

make_env()

Return a dict of environment variables to use for prepare/run.

By default, make_env() will return:

{
    "MILABENCH_DIR_CODE": self.dirs.code,
    "MILABENCH_DIR_DATA": self.dirs.data,
    "MILABENCH_DIR_VENV": self.dirs.venv,
    "MILABENCH_DIR_RUNS": self.dirs.runs,
    "MILABENCH_CONFIG": json.dumps(self.config),
}

async pin(clear_previous: bool = True, pip_compile_args: Sequence = (), input_files: Sequence = (), constraints: Sequence = (), working_dir=None, requirements: Sequence = ())

Pin versions to requirements file.

Parameters:

*pip_compile_args – python3 -m piptools compile extra arguments
requirements_file – The output requirements file
input_files – A list of inputs to piptools compile
constraint – The constraint file

async prepare()

Prepare the benchmark.

By default, this executes self.dirs.code / self.prepare_script, which should be an executable (python, bash, etc.)

The environment variables from make_env() are set for that invocation, so the script can use e.g. $MILABENCH_DIR_DATA to access the data directory for the benchmark.

The default value of self.prepare_script is "prepare.py".

resolve_argument(name, default): Resolve as single placeholder argument

resolve_placeholder(placeholder)

Resolve as single placeholder argument

Examples

>>> resolve_placeholder("auto({n_worker}, 8)")
16

async run()

Start the benchmark and return the running process.

By default, this runs:

voir {*voirargs} {self.dirs.code / self.main_script} {*args}

The environment variables from make_env() are set for that invocation, so the script can use e.g. $MILABENCH_DIR_DATA to access the data directory for the benchmark.

The default value of self.main_script is "main.py".

Parameters:

args – A list of arguments to the program.
voirargs – A list of arguments to voir.
env – Environment variables to set for the process.